Re: Defining AVERAGING clause

[Robert Goldman]

On 14 Mar 2018, at 11:13, Russ Tyndall wrote:

I think the natural ordering of things means this wont be a problem in 
99% of situations. Obviously you cant use mean until the finally 
clause, but all of your finally's should come after the averaging 
finally.


Agreed.  Still, it's kind of an unhappy feeling to be relying on this 
not very user-controlled behavior, and if some day downstream it goes 
wrong, that would be problematic.


I guess one would want to be able to build `finally` and `initially` 
clauses with `defmacro-clause` where you could specify whether they come 
before all end-programmer clauses of the same type, or before all such 
clauses of the same type.  I suppose one could add sorting to the 
`final-code` and `init-code` in `iter`, but that seems fraught with 
peril.  You're probably right that I should just not worry about this.




(iterate:defmacro-clause (averaging expr  into var)
  (let ((avg (or var iterate::*result-var*)))
(alexandria:with-unique-names (cnt sum)
  `(progn
(with ,avg)
(with ,sum = 0)
(with ,cnt = 0)
(incf ,cnt)
(incf ,sum ,expr)

(finally
 (setf ,avg (if (plusp ,cnt) (/ ,sum ,cnt) 0))
 )

(iter (for i from 0 to 100)
  (averaging i into mean)
  (finally (return mean)))
=> 50

Expands to:
(LET* ((I NIL) (MEAN NIL) (#:SUM4008 0) (#:CNT4007 0))
  (BLOCK NIL
(TAGBODY
  (PROGN (SETQ I -1))
 LOOP-TOP-NIL
  (PROGN
   (SETQ I (+ I 1))
   (IF (> I 100)
   (GO LOOP-END-NIL))
   (PROGN
(SETQ #:CNT4007 (+ 1 #:CNT4007))
(SETQ #:SUM4008 (+ I #:SUM4008
  (PROGN)
  (GO LOOP-TOP-NIL)
 LOOP-END-NIL
  (PROGN
   (SETF MEAN
   (IF (PLUSP #:CNT4007)
   (/ #:SUM4008 #:CNT4007)
   0))
   (RETURN MEAN)))
NIL))

Cheers,
Russ Tyndall
Acceleration.net


- Original Message -
From: Robert Goldman [mailto:rpgold...@sift.info]
To: r...@acceleration.net
Cc: iterate-devel@common-lisp.net
Sent: Tue, 13 Mar 2018 12:27:47 -0500
Subject: Re: Defining AVERAGING clause

Wouldn't this technique have trouble with code movement issues?  For
example, if I have
```
(iter (for x in data-table)
(averaging x into mean)
(finally (return mean)))
```
I know that's a contrived example, but I think the point is clear: if
I'm going to make `averaging` a collector, I'd like to be able to use
its result, and if that result is computed in a `finally` block that's
invisible and out of my control, I can't do it, can I?  There's
`finally-protected`, but I think that's the *opposite* of what I need.

Best,
r


On 13 Mar 2018, at 10:08, Russ Tyndall wrote:


Here is an existing "sampling" clause to pull a random sample from a
larger data set.  The long and short is just use a finally clause, 
as

you would when writing a normal iterate loop.

(iterate:defmacro-clause (sampling expr  into var size size)
  "resevoir sample the input"
  (let ((sample (or var iterate::*result-var*)))
    (alexandria:with-unique-names (i sample-size sigil buffer row)
  `(progn
    (with ,sample)
    (with ,sample-size = (or ,size 100))
    (with ,buffer = (make-array ,sample-size
:initial-element ',sigil))
    (with ,i = 0)
    (if (< ,i ,sample-size)
    (setf (aref ,buffer ,i) ,expr)
    (let ((r (random ,i)))
  (when (< r ,sample-size)
    (setf (aref ,buffer r) ,expr
    (incf ,i)
    (finally
 ;; convert our sample to a list, but only if we
actually took the sample
 (when (plusp ,i)
   (setf ,sample
 (iter (for ,row in-vector ,buffer)
   (until (eq ,row ',sigil))
   (collect ,row)

Cheers,
Russ Tyndall
Acceleration.net
On 03/13/2018 10:49 AM, Robert Goldman wrote:


I was going to define an |AVERAGING| collector clause for iterate,
but I'm not sure how to do it. The obvious thing, it seemed to me,
would be to sum the values as I go along, and count them, and then
divide the sum by the count when leaving the loop.

But the examples for |DEFMACRO-CLAUSE| in the manual do all of their
work while iterating, and there doesn't seem to be an "at-end" hook.
Is the kind of thing I would like feasible, and if so, how is it to
be done?

thanks!
r

Re: Defining AVERAGING clause

[Russ Tyndall]

I think the natural ordering of things means this wont be a problem in 99% of 
situations. Obviously you cant use mean until the finally clause, but all of 
your finally's should come after the averaging finally.

(iterate:defmacro-clause (averaging expr  into var)
  (let ((avg (or var iterate::*result-var*)))
(alexandria:with-unique-names (cnt sum)
  `(progn
(with ,avg)
(with ,sum = 0)
(with ,cnt = 0)
(incf ,cnt)
(incf ,sum ,expr)

(finally
 (setf ,avg (if (plusp ,cnt) (/ ,sum ,cnt) 0))
 )

(iter (for i from 0 to 100)
  (averaging i into mean)
  (finally (return mean)))
=> 50

Expands to:
(LET* ((I NIL) (MEAN NIL) (#:SUM4008 0) (#:CNT4007 0))
  (BLOCK NIL
(TAGBODY
  (PROGN (SETQ I -1))
 LOOP-TOP-NIL
  (PROGN
   (SETQ I (+ I 1))
   (IF (> I 100)
   (GO LOOP-END-NIL))
   (PROGN
(SETQ #:CNT4007 (+ 1 #:CNT4007))
(SETQ #:SUM4008 (+ I #:SUM4008
  (PROGN)
  (GO LOOP-TOP-NIL)
 LOOP-END-NIL
  (PROGN
   (SETF MEAN
   (IF (PLUSP #:CNT4007)
   (/ #:SUM4008 #:CNT4007)
   0))
   (RETURN MEAN)))
NIL))

Cheers,
Russ Tyndall
Acceleration.net


- Original Message -
From: Robert Goldman [mailto:rpgold...@sift.info]
To: r...@acceleration.net
Cc: iterate-devel@common-lisp.net
Sent: Tue, 13 Mar 2018 12:27:47 -0500
Subject: Re: Defining AVERAGING clause

Wouldn't this technique have trouble with code movement issues?  For
example, if I have
```
(iter (for x in data-table)
(averaging x into mean)
(finally (return mean)))
```
I know that's a contrived example, but I think the point is clear: if
I'm going to make `averaging` a collector, I'd like to be able to use
its result, and if that result is computed in a `finally` block that's
invisible and out of my control, I can't do it, can I?  There's
`finally-protected`, but I think that's the *opposite* of what I need.

Best,
r


On 13 Mar 2018, at 10:08, Russ Tyndall wrote:

> Here is an existing "sampling" clause to pull a random sample from a
> larger data set.  The long and short is just use a finally clause, as
> you would when writing a normal iterate loop.
>
> (iterate:defmacro-clause (sampling expr  into var size size)
>   "resevoir sample the input"
>   (let ((sample (or var iterate::*result-var*)))
>     (alexandria:with-unique-names (i sample-size sigil buffer row)
>   `(progn
>     (with ,sample)
>     (with ,sample-size = (or ,size 100))
>     (with ,buffer = (make-array ,sample-size
> :initial-element ',sigil))
>     (with ,i = 0)
>     (if (< ,i ,sample-size)
>     (setf (aref ,buffer ,i) ,expr)
>     (let ((r (random ,i)))
>   (when (< r ,sample-size)
>     (setf (aref ,buffer r) ,expr
>     (incf ,i)
>     (finally
>  ;; convert our sample to a list, but only if we
> actually took the sample
>  (when (plusp ,i)
>    (setf ,sample
>  (iter (for ,row in-vector ,buffer)
>    (until (eq ,row ',sigil))
>    (collect ,row)
>
> Cheers,
> Russ Tyndall
> Acceleration.net
> On 03/13/2018 10:49 AM, Robert Goldman wrote:
>>
>> I was going to define an |AVERAGING| collector clause for iterate,
>> but I'm not sure how to do it. The obvious thing, it seemed to me,
>> would be to sum the values as I go along, and count them, and then
>> divide the sum by the count when leaving the loop.
>>
>> But the examples for |DEFMACRO-CLAUSE| in the manual do all of their
>> work while iterating, and there doesn't seem to be an "at-end" hook.
>> Is the kind of thing I would like feasible, and if so, how is it to
>> be done?
>>
>> thanks!
>> r
>>

Re: Defining AVERAGING clause

[Robert Goldman]

Wouldn't this technique have trouble with code movement issues?  For 
example, if I have

```
(iter (for x in data-table)
   (averaging x into mean)
   (finally (return mean)))
```
I know that's a contrived example, but I think the point is clear: if 
I'm going to make `averaging` a collector, I'd like to be able to use 
its result, and if that result is computed in a `finally` block that's 
invisible and out of my control, I can't do it, can I?  There's 
`finally-protected`, but I think that's the *opposite* of what I need.


Best,
r


On 13 Mar 2018, at 10:08, Russ Tyndall wrote:

Here is an existing "sampling" clause to pull a random sample from a 
larger data set.  The long and short is just use a finally clause, as 
you would when writing a normal iterate loop.


(iterate:defmacro-clause (sampling expr  into var size size)
  "resevoir sample the input"
  (let ((sample (or var iterate::*result-var*)))
    (alexandria:with-unique-names (i sample-size sigil buffer row)
  `(progn
    (with ,sample)
    (with ,sample-size = (or ,size 100))
    (with ,buffer = (make-array ,sample-size 
:initial-element ',sigil))

    (with ,i = 0)
    (if (< ,i ,sample-size)
    (setf (aref ,buffer ,i) ,expr)
    (let ((r (random ,i)))
  (when (< r ,sample-size)
    (setf (aref ,buffer r) ,expr
    (incf ,i)
    (finally
 ;; convert our sample to a list, but only if we 
actually took the sample

 (when (plusp ,i)
   (setf ,sample
 (iter (for ,row in-vector ,buffer)
   (until (eq ,row ',sigil))
   (collect ,row)

Cheers,
Russ Tyndall
Acceleration.net
On 03/13/2018 10:49 AM, Robert Goldman wrote:


I was going to define an |AVERAGING| collector clause for iterate, 
but I'm not sure how to do it. The obvious thing, it seemed to me, 
would be to sum the values as I go along, and count them, and then 
divide the sum by the count when leaving the loop.


But the examples for |DEFMACRO-CLAUSE| in the manual do all of their 
work while iterating, and there doesn't seem to be an "at-end" hook. 
Is the kind of thing I would like feasible, and if so, how is it to 
be done?


thanks!
r

Re: Defining AVERAGING clause

[Luís Oliveira]

I shouldn't have used mapc as an example. Consider instead some random,
opaque, map-like function that couldn't be rewritten into an iter form. In
any case, your collectors library seems interesting. Will check it out,
thanks!

On Tue, Mar 13, 2018 at 4:08 PM Russ Tyndall  wrote:

> Here are two different ways of rewriting that mapc with iter:
>
> `(iter top (repeat 1) (iter (for x in '(1 2 3)) (in top (collect x`
>
> `(iter (repeat 1) (appending '(1 2 3)))`
>
> `(iter (repeat 1) (appending (iter (for x in '(1 2 3)) (collect x`
>
> If I am using iter, I try hard to *only* use iter, as one of the reasons
> for this library (for me) is to have a single syntax for iteration.
>
> That said, I wrote the `collectors` library to handle arbitrary collection
> tasks that were not easily expressed as a straight iteration.
> The collectors library binds local functions, so you can `(mapc #'collect
> ...)` etc.
>
> https://github.com/AccelerationNet/collectors
>
> Cheers,
>
> Russ Tyndall
>
> On 03/13/2018 11:52 AM, Luís Oliveira wrote:
>
> Just a curiosity: you could compute the average incrementally: <
> https://math.stackexchange.com/questions/106700/incremental-averageing>,
> but doing it the usual way is probably more efficient and with smaller
> numeric error when using floats.
>
> Slightly off-topic question: oftentimes I have to use map-like iterators.
> Is there a good way to use such iterators in conjunction with iter? An
> obvious way would be something like:
>
> (iter (repeat 1) (mapc (lambda (x) (collect x)) '(1 2 3)))
>
> The (iter (repeat 1) ...) bit could be hidden under a macro, I suppose.
> But it'd be nicer if collect were a function, so I could do (mapc #'collect
> '(1 2 3)). Perhaps said macro could rewrite #'clause to (lambda (x)
> (expansion-of-clause x)) or something similar. Looks like a fun idea to
> explore so I'm wondering if someone's played with something like this
> before.
>
> (It's even more fun to turn map-like iterators into generators using
> continuations, but that's fully off-topic. :-))
>
> Cheers,
> Luís
>
> On Tue, Mar 13, 2018 at 3:08 PM Russ Tyndall 
> wrote:
>
>> Here is an existing "sampling" clause to pull a random sample from a
>> larger data set.  The long and short is just use a finally clause, as you
>> would when writing a normal iterate loop.
>>
>> (iterate:defmacro-clause (sampling expr  into var size size)
>>   "resevoir sample the input"
>>   (let ((sample (or var iterate::*result-var*)))
>> (alexandria:with-unique-names (i sample-size sigil buffer row)
>>   `(progn
>> (with ,sample)
>> (with ,sample-size = (or ,size 100))
>> (with ,buffer = (make-array ,sample-size :initial-element
>> ',sigil))
>> (with ,i = 0)
>> (if (< ,i ,sample-size)
>> (setf (aref ,buffer ,i) ,expr)
>> (let ((r (random ,i)))
>>   (when (< r ,sample-size)
>> (setf (aref ,buffer r) ,expr
>> (incf ,i)
>> (finally
>>  ;; convert our sample to a list, but only if we actually took
>> the sample
>>  (when (plusp ,i)
>>(setf ,sample
>>  (iter (for ,row in-vector ,buffer)
>>(until (eq ,row ',sigil))
>>(collect ,row)
>>
>> Cheers,
>> Russ Tyndall
>> Acceleration.net
>>
>> On 03/13/2018 10:49 AM, Robert Goldman wrote:
>>
>> I was going to define an AVERAGING collector clause for iterate, but I'm
>> not sure how to do it. The obvious thing, it seemed to me, would be to sum
>> the values as I go along, and count them, and then divide the sum by the
>> count when leaving the loop.
>>
>> But the examples for DEFMACRO-CLAUSE in the manual do all of their work
>> while iterating, and there doesn't seem to be an "at-end" hook. Is the kind
>> of thing I would like feasible, and if so, how is it to be done?
>>
>> thanks!
>> r
>>
>>
>>
>

Re: Defining AVERAGING clause

[Russ Tyndall]

Here are two different ways of rewriting that mapc with iter:

`(iter top (repeat 1) (iter (for x in '(1 2 3)) (in top (collect x`

`(iter (repeat 1) (appending '(1 2 3)))`

`(iter (repeat 1) (appending (iter (for x in '(1 2 3)) (collect x`

If I am using iter, I try hard to *only* use iter, as one of the reasons 
for this library (for me) is to have a single syntax for iteration.


That said, I wrote the `collectors` library to handle arbitrary 
collection tasks that were not easily expressed as a straight iteration.
The collectors library binds local functions, so you can `(mapc 
#'collect ...)` etc.


https://github.com/AccelerationNet/collectors

Cheers,

Russ Tyndall


On 03/13/2018 11:52 AM, Luís Oliveira wrote:
Just a curiosity: you could compute the average incrementally: 
, 
but doing it the usual way is probably more efficient and with smaller 
numeric error when using floats.


Slightly off-topic question: oftentimes I have to use map-like 
iterators. Is there a good way to use such iterators in conjunction 
with iter? An obvious way would be something like:


    (iter (repeat 1) (mapc (lambda (x) (collect x)) '(1 2 3)))

The (iter (repeat 1) ...) bit could be hidden under a macro, I 
suppose. But it'd be nicer if collect were a function, so I could do 
(mapc #'collect '(1 2 3)). Perhaps said macro could rewrite #'clause 
to (lambda (x) (expansion-of-clause x)) or something similar. Looks 
like a fun idea to explore so I'm wondering if someone's played with 
something like this before.


(It's even more fun to turn map-like iterators into generators using 
continuations, but that's fully off-topic. :-))


Cheers,
Luís

On Tue, Mar 13, 2018 at 3:08 PM Russ Tyndall > wrote:


Here is an existing "sampling" clause to pull a random sample from
a larger data set.  The long and short is just use a finally
clause, as you would when writing a normal iterate loop.

(iterate:defmacro-clause (sampling expr  into var size size)
  "resevoir sample the input"
  (let ((sample (or var iterate::*result-var*)))
    (alexandria:with-unique-names (i sample-size sigil buffer row)
  `(progn
    (with ,sample)
    (with ,sample-size = (or ,size 100))
    (with ,buffer = (make-array ,sample-size :initial-element
',sigil))
    (with ,i = 0)
    (if (< ,i ,sample-size)
    (setf (aref ,buffer ,i) ,expr)
    (let ((r (random ,i)))
  (when (< r ,sample-size)
    (setf (aref ,buffer r) ,expr
    (incf ,i)
    (finally
 ;; convert our sample to a list, but only if we actually
took the sample
 (when (plusp ,i)
   (setf ,sample
 (iter (for ,row in-vector ,buffer)
   (until (eq ,row ',sigil))
   (collect ,row)

Cheers,
Russ Tyndall
Acceleration.net

On 03/13/2018 10:49 AM, Robert Goldman wrote:


I was going to define an |AVERAGING| collector clause for
iterate, but I'm not sure how to do it. The obvious thing, it
seemed to me, would be to sum the values as I go along, and count
them, and then divide the sum by the count when leaving the loop.

But the examples for |DEFMACRO-CLAUSE| in the manual do all of
their work while iterating, and there doesn't seem to be an
"at-end" hook. Is the kind of thing I would like feasible, and if
so, how is it to be done?

thanks!
r

Defining AVERAGING clause

[Robert Goldman]

I was going to define an `AVERAGING` collector clause for iterate, but 
I'm not sure how to do it.  The obvious thing, it seemed to me, would be 
to sum the values as I go along, and count them, and then divide the sum 
by the count when leaving the loop.


But the examples for `DEFMACRO-CLAUSE` in the manual do all of their 
work while iterating, and there doesn't seem to be an "at-end" hook.  Is 
the kind of thing I would like feasible, and if so, how is it to be 
done?


thanks!
r