Re: [Factor-talk] Questions of a newcomer

[petern]

Hi Chris,

On 2016-11-10 11:03, Chris Double wrote:
> With that I see what you are seeing. Both numbers print out after the
> last thread finishes. It looks like it's buffering in this case. If I
> add a 'flush' then I see them printed after 5 seconds then 10 seconds:
> 
> -8<-
> USING: fry prettyprint kernel io namespaces sequences io.launcher
> io.directories io.encodings.utf8 io.files io.files.info io.pathnames
> concurrency.messaging threads math tools.threads accessors calendar ;
> IN: script
> 
> self '[ "bash -c \"sleep 10\"" run-process drop 1 _ send ] "1" spawn
> self '[ "bash -c \"sleep 5\"" run-process drop 2 _ send ] "2" spawn
> receive . flush
> receive . flush
> clear
> -8<-
> 
> Could this fix your issue?

Indeed that's it. Thanks for your help, I should be able to finish it 
from this point :)

-- 

   Peter Nagy


--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[Chris Double]

On Thu, Nov 10, 2016 at 10:00 PM,   wrote:
> Any
> more ideas why? Is run-process blocking everyone? Is there some FFI call
> like you mentioned? Where could I start to debug this on my own?

The only difference to what you are doing and what my test did was
you're running as a script. So I tried this:

-8<-
USING: fry prettyprint kernel io namespaces sequences io.launcher
io.directories io.encodings.utf8 io.files io.files.info io.pathnames
concurrency.messaging threads math tools.threads accessors calendar ;
IN: script

self '[ "bash -c \"sleep 10\"" run-process drop 1 _ send ] "1" spawn
self '[ "bash -c \"sleep 5\"" run-process drop 2 _ send ] "2" spawn
receive .
receive .
clear
-8<-


With that I see what you are seeing. Both numbers print out after the
last thread finishes. It looks like it's buffering in this case. If I
add a 'flush' then I see them printed after 5 seconds then 10 seconds:

-8<-
USING: fry prettyprint kernel io namespaces sequences io.launcher
io.directories io.encodings.utf8 io.files io.files.info io.pathnames
concurrency.messaging threads math tools.threads accessors calendar ;
IN: script

self '[ "bash -c \"sleep 10\"" run-process drop 1 _ send ] "1" spawn
self '[ "bash -c \"sleep 5\"" run-process drop 2 _ send ] "2" spawn
receive . flush
receive . flush
clear
-8<-

Could this fix your issue?

-- 
http://bluishcoder.co.nz

--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[petern]

Hi John,

On 2016-11-08 22:38, John Benediktsson wrote:
> There are a lot of ways to solve the problem, but without knowing more
> about what you're looking for, I'll just leave these here.

I spent the last half hour reading your code, running it and examining 
the vocabularies. Very helpful, thank you for your time!

-- 

   Peter Nagy


--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[petern]

Hi Chris,

On 2016-11-08 23:58, Chris Double wrote:
> I tried to duplicate the basics of your code with the following:
> 
> self '[ "bash -c \"sleep 10\"" run-process drop 1 _ send ] "1" spawn
> self '[ "bash -c \"sleep 5\"" run-process drop 2 _ send ] "2" spawn
> receive
> 
> This will spawn two threads that run a bash 'sleep'. The 'receive'
> blocks until the shortest sleep finishes and if you run another
> 'receive' then it blocks until the longer one completes. This should
> be what your code does too. Are you sure the git commands aren't all
> completing at the same time?

After a simple test you are right, the threads send the message at the 
same time. I'm still left wondering why. There's e.g. 10 threads running 
a `(thread on stack here) "git pull" run-process "pulled" send`. Now the 
git output comes to stdout intermixed and only after all threads 
finished with the run-process do they get time to send the message. Any 
more ideas why? Is run-process blocking everyone? Is there some FFI call 
like you mentioned? Where could I start to debug this on my own?

> 
> Note the use of the 'fry' quotation to avoid having to curry and swap
> later to pass the 'self' around btw.

Yes looks much cleaner, thanks. I am avoiding fry and locals and similar 
helpers for now to train the shuffling/composing more.

Thank you for your answer, much appreciated.

-- 

   Peter Nagy


--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[Chris Double]

On Wed, Nov 9, 2016 at 4:02 AM,   wrote:
>
> There are my last 2 attemps. The first, commented out version finishes
> without waiting for the threads to finish (even with the ugly hack of
> reading the state>> of the thread) while in the second the receiving
> thread doesn't read the messages as they arrive, rather its mailbox gets
> filled up and only when everyone finishes gets time to work on the
> messages. What am I doing wrong?

Nothing seems to be wrong here, but there's no obvious thread join
operation to wait on completion so Factor exits immediately even if
active threads are running.

> On the topic of factor's cooperative threads - do they run multi- or
> single-core?

The are single core co-operative threads. If you run an FFI function
that  blocks then all threads block.

> If you have some other tips on the code I'll be glad, I feel like I'm
> doing more shuffling then I might need.

I tried to duplicate the basics of your code with the following:

self '[ "bash -c \"sleep 10\"" run-process drop 1 _ send ] "1" spawn
self '[ "bash -c \"sleep 5\"" run-process drop 2 _ send ] "2" spawn
receive

This will spawn two threads that run a bash 'sleep'. The 'receive'
blocks until the shortest sleep finishes and if you run another
'receive' then it blocks until the longer one completes. This should
be what your code does too. Are you sure the git commands aren't all
completing at the same time?

Note the use of the 'fry' quotation to avoid having to curry and swap
later to pass the 'self' around btw.

-- 
http://bluishcoder.co.nz

--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[John Benediktsson]

>
> I thought of parallel-map first but I wanted to do a bit more than that,
> otherwise I could just write a script that handles one and feed it to
> GNU parallel. I wanted to achieve more than what I can with GNU parallel
> (which is able to parallelize on multiple cores and writes correctly to
> stdout by not mixing the outputs of several tasks). Since I know the
> count ahead I wanted to suppress the git output and instead show a
> progress bar with some results, e.g. if the pull/fetch was successful or
> not. The only way I can imagine that is by message passing. But as I
> explained the threads don't seem to work as I expect, the recieving
> thread doesn't get to "speak up" until it's too long. Is there any way
> one can make a particular thread higher priority or come up right when
> it gets a message? Also, when does factor end? I have a bunch of threads
> that are waiting for I/O or network response and it finishes anyway. My
> problem with threads is that I don't understand what's happening behind
> the scenes, who goes when, how many of them at once, etc.


Here is a simple example of ``n`` worker threads that do some number of
work (in this case sleeping and then doing "1" unit of work) and then push
that work into a mailbox.  The main thread does a mailbox-get and adds that
work to its total and updates a progress-bar, finishing when all the
expected work is done:

:: start-thread-example1 ( n -- )

 :> box

n iota [
'[
_ 100 * milliseconds sleep
1 box mailbox-put
] "Worker" spawn drop
] each

0 [
box mailbox-get +
dup n / 70 make-progress-bar print
dup n <
] loop drop ;


In the ``progress-bars.model`` vocabulary we have a visual model that can
be used in a GUI, or you can just print it to standard-out.

If you wanted to see how the count-downs code might look (without
progress-bars), maybe something like this:

:: start-thread-example2 ( n -- )

n  :> coord

n iota [
'[
_ 100 * milliseconds sleep
coord count-down
] "Worker" spawn drop
] each

coord await ;

There are a lot of ways to solve the problem, but without knowing more
about what you're looking for, I'll just leave these here.

Best,
John.
--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[petern]

Hi John,

On 2016-11-08 16:52, John Benediktsson wrote:
>> 
>> In that case I guess the mailing list makes more sense. Unless there's
>> people reading the IRC logs and not part of the mailing list.
>> 
> 
> The mailing list can be a fine place, or GitHub issues if you run into 
> some
> problems with Factor.  If you are worried about higher volume of
> conversation, you could reach out to me and we could have an offline
> discussion, pulling a couple of the other developers into it as well.

I'm OK with the mailing list, but let me know if you want to take the 
conversation off of it.

> 
> 
>> Thanks, I guess your explanation makes sense, `[ y ] dip` looked a bit
>> weird to me on the first look but I understand the meaning now.
>> 
> 
> There should not be a performance difference, in a simple example of `[ 
> y ]
> dip` versus `y swap``, its based on a sense of which one looks cleaner 
> or
> more closely expresses the purpose of the code.  You can find other 
> similar
> examples, like `[ foo ] keep` versus `[ foo ] [ ] bi` where developers 
> are
> experimenting with and thinking about expressivity in concatenative
> languages.

OK, guess I'll experiment as well, thanks.

> 
> 
>> > I've seen discussions on this mailing list about the extra/cursors
>> > vocabulary about that. I've never used it though. For example Joe
>> > talked
>> > about it in 2009:
>> > https://sourceforge.net/p/factor/mailman/message/23878123
>> >
>> 
>> I see, so nothing that would be alive.
>> 
> 
> Well, possibly thats one way to look at it.  Another is that it is/was 
> an
> experiment in thinking about generalized iterators and has tests and 
> works
> fine for what it is.  You can see the code in extra/cursors or use it
> (`USE: cursors`).  We do need to move towards lazy/infinite
> generator/iterators and hopefully to more easily generalize
> iteration/operations on those as well as sequences (`each`) and assocs
> (`assoc-each`) and dlists (`dlist-each`) and lists (`leach`) and heaps,
> trees, etc etc.  Doug Coleman did some work on this, but we haven't
> finished it yet.

I looked into the cursors vocab, sadly it doesn't have any 
documentation. But I'll dig into it if the need arises.

> 
> In the meantime I finished a small script where I was trying out factor
>> - it goes into my git folders and fetches/pulls updates from origin. I
>> got the first version working pretty quickly, then I wanted to tweak 
>> it
>> so that the updates run in parallel and output updates on the console
>> live. In a final version I wanted to print some progressbar while
>> surpressing git output. For that reason I thought I need message
>> passing. This isn't working as I expect it to. Here is the gist:
>> 
>> https://gist.github.com/xificurC/f4de1993b3218a50dd8936dfc0ec16f2
>> 
>> There are my last 2 attemps. The first, commented out version finishes
>> without waiting for the threads to finish (even with the ugly hack of
>> reading the state>> of the thread) while in the second the receiving
>> thread doesn't read the messages as they arrive, rather its mailbox 
>> gets
>> filled up and only when everyone finishes gets time to work on the
>> messages. What am I doing wrong?
>> 
> 
> I would think it would be easier to use some form of ``parallel-map`` 
> from
> `USE: concurrency.combinators`.  But if you want to build it up from 
> other
> concurrency libraries, you might think of a count-down latch (`USE:
> concurrency.count-downs`) and waiting for it to reach zero, since you 
> know
> how many items to wait for.

I thought of parallel-map first but I wanted to do a bit more than that, 
otherwise I could just write a script that handles one and feed it to 
GNU parallel. I wanted to achieve more than what I can with GNU parallel 
(which is able to parallelize on multiple cores and writes correctly to 
stdout by not mixing the outputs of several tasks). Since I know the 
count ahead I wanted to suppress the git output and instead show a 
progress bar with some results, e.g. if the pull/fetch was successful or 
not. The only way I can imagine that is by message passing. But as I 
explained the threads don't seem to work as I expect, the recieving 
thread doesn't get to "speak up" until it's too long. Is there any way 
one can make a particular thread higher priority or come up right when 
it gets a message? Also, when does factor end? I have a bunch of threads 
that are waiting for I/O or network response and it finishes anyway. My 
problem with threads is that I don't understand what's happening behind 
the scenes, who goes when, how many of them at once, etc.

>  You might also want to limit the number of
> threads running by using a semaphore (`USE: concurrency.semaphores`) if 
> not
> using parallel-map on groups of items.

Would that help to keep the receiver alive? Like if there's a total of 4 
threads running I put a semaphore on 3 and I get one always through? The 
problem with this example concretely is that I'm waiting on 

Re: [Factor-talk] Questions of a newcomer

[John Benediktsson]

>
> In that case I guess the mailing list makes more sense. Unless there's
> people reading the IRC logs and not part of the mailing list.
>

The mailing list can be a fine place, or GitHub issues if you run into some
problems with Factor.  If you are worried about higher volume of
conversation, you could reach out to me and we could have an offline
discussion, pulling a couple of the other developers into it as well.


> Thanks, I guess your explanation makes sense, `[ y ] dip` looked a bit
> weird to me on the first look but I understand the meaning now.
>

There should not be a performance difference, in a simple example of `[ y ]
dip` versus `y swap``, its based on a sense of which one looks cleaner or
more closely expresses the purpose of the code.  You can find other similar
examples, like `[ foo ] keep` versus `[ foo ] [ ] bi` where developers are
experimenting with and thinking about expressivity in concatenative
languages.


> > I've seen discussions on this mailing list about the extra/cursors
> > vocabulary about that. I've never used it though. For example Joe
> > talked
> > about it in 2009:
> > https://sourceforge.net/p/factor/mailman/message/23878123
> >
>
> I see, so nothing that would be alive.
>

Well, possibly thats one way to look at it.  Another is that it is/was an
experiment in thinking about generalized iterators and has tests and works
fine for what it is.  You can see the code in extra/cursors or use it
(`USE: cursors`).  We do need to move towards lazy/infinite
generator/iterators and hopefully to more easily generalize
iteration/operations on those as well as sequences (`each`) and assocs
(`assoc-each`) and dlists (`dlist-each`) and lists (`leach`) and heaps,
trees, etc etc.  Doug Coleman did some work on this, but we haven't
finished it yet.

In the meantime I finished a small script where I was trying out factor
> - it goes into my git folders and fetches/pulls updates from origin. I
> got the first version working pretty quickly, then I wanted to tweak it
> so that the updates run in parallel and output updates on the console
> live. In a final version I wanted to print some progressbar while
> surpressing git output. For that reason I thought I need message
> passing. This isn't working as I expect it to. Here is the gist:
>
> https://gist.github.com/xificurC/f4de1993b3218a50dd8936dfc0ec16f2
>
> There are my last 2 attemps. The first, commented out version finishes
> without waiting for the threads to finish (even with the ugly hack of
> reading the state>> of the thread) while in the second the receiving
> thread doesn't read the messages as they arrive, rather its mailbox gets
> filled up and only when everyone finishes gets time to work on the
> messages. What am I doing wrong?
>

I would think it would be easier to use some form of ``parallel-map`` from
`USE: concurrency.combinators`.  But if you want to build it up from other
concurrency libraries, you might think of a count-down latch (`USE:
concurrency.count-downs`) and waiting for it to reach zero, since you know
how many items to wait for.  You might also want to limit the number of
threads running by using a semaphore (`USE: concurrency.semaphores`) if not
using parallel-map on groups of items.


> On the topic of factor's cooperative threads - do they run multi- or
> single-core?
>

Currently single-core.  We have some work completed to allow multiple
cooperative "Factor VM's" in a single process but it's not finished.


> If you have some other tips on the code I'll be glad, I feel like I'm
> doing more shuffling then I might need.
>

Often there is a combinator that can make your operation simpler, or a
different order to the stack that flows through your operations more
simply.  But feel free to use local variables when you are learning and
having a hard time with stack shuffling.  They don't come with a
performance cost (except mutable variables where you are changing the value
pointed to by the local name).  And keep asking questions!

Best,
John.
--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[petern]

Hi Jon,

On 2016-11-07 22:34, Jon Harper wrote:
> Hi Peter,
> 
> On Mon, Nov 7, 2016 at 3:07 PM,  wrote:
> 
>> Hello,
>> 
>> I am tinkering with factor and was wondering if it is OK to pick your
>> brains here? As I play around with the language questions come up that
>> are probably easy for you to answer. I don't see much action on the
>> #concatenative IRC channel so I thought the maling list might be a
>> better place?
>> 
>> Questions are always welcome. The mailing list or #concatenative on 
>> IRC
> are both good places to ask. Several people read the logs of 
> #concatenative
> and may answer your questions some time after you sent it, so don't 
> give up.

In that case I guess the mailing list makes more sense. Unless there's 
people reading the IRC logs and not part of the mailing list.

> 
> As a starter:
>> 
>> - I see a common pattern in definitions of using `dip` instead of
>> `swap`. Is there some special reason for that? Is it more performant? 
>> I
>> know the words aren't interchangeable but e.g. `with-directory-files`
>> has `[ [ "" directory-files ] ] dip` which as far as I can tell is
>> equivalent to `[ "" directory-files ] swap`. I saw this pattern in 
>> more
>> definitions.
>> 
> I guess it's a matter of personal style. I would argue that the 
> 'meaning'
> of swap ( x y -- y x ) is that there's the same importance on pushing y
> down the stack than on puling x up the stack. Whereas [ y ] dip would 
> focus
> more putting y down the stack.
> 
> Regarding the performance, you can often see for yourself using the
> optimized. word of compiler.tree.debugger. You could even install 
> libudis
> and use the disassemble word of tools.disassembler.
> I would be surprised if the performance of swap vs dip mattered in a 
> real
> application.
> 

Thanks, I guess your explanation makes sense, `[ y ] dip` looked a bit 
weird to me on the first look but I understand the meaning now.

> 
>> 
>> - is there any sequence generator/iterator vocabulary? Something that
>> gives or computes values on demand. One can find it in many languages
>> with a bit different flavor, e.g. Scheme, Rust, Python. I saw that 
>> there
>> is lists.lazy which can serve a similar purpose but is a bit more 
>> heavy
>> weight in some cases. Maybe this isn't needed in factor at all and you
>> use a different pattern to solve a similar problem?
>> 
> I've seen discussions on this mailing list about the extra/cursors
> vocabulary about that. I've never used it though. For example Joe 
> talked
> about it in 2009: 
> https://sourceforge.net/p/factor/mailman/message/23878123
> 

I see, so nothing that would be alive.



In the meantime I finished a small script where I was trying out factor 
- it goes into my git folders and fetches/pulls updates from origin. I 
got the first version working pretty quickly, then I wanted to tweak it 
so that the updates run in parallel and output updates on the console 
live. In a final version I wanted to print some progressbar while 
surpressing git output. For that reason I thought I need message 
passing. This isn't working as I expect it to. Here is the gist:

https://gist.github.com/xificurC/f4de1993b3218a50dd8936dfc0ec16f2

There are my last 2 attemps. The first, commented out version finishes 
without waiting for the threads to finish (even with the ugly hack of 
reading the state>> of the thread) while in the second the receiving 
thread doesn't read the messages as they arrive, rather its mailbox gets 
filled up and only when everyone finishes gets time to work on the 
messages. What am I doing wrong?

On the topic of factor's cooperative threads - do they run multi- or 
single-core?

If you have some other tips on the code I'll be glad, I feel like I'm 
doing more shuffling then I might need.

Thank you for your answer.

> Cheers,
> Jon
> 
> --
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> ___
> Factor-talk mailing list
> Factor-talk@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/factor-talk

-- 

   Peter Nagy


--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk

Re: [Factor-talk] Questions of a newcomer

[Jon Harper]

Hi Peter,

On Mon, Nov 7, 2016 at 3:07 PM,  wrote:

> Hello,
>
> I am tinkering with factor and was wondering if it is OK to pick your
> brains here? As I play around with the language questions come up that
> are probably easy for you to answer. I don't see much action on the
> #concatenative IRC channel so I thought the maling list might be a
> better place?
>
> Questions are always welcome. The mailing list or #concatenative on IRC
are both good places to ask. Several people read the logs of #concatenative
and may answer your questions some time after you sent it, so don't give up.

As a starter:
>
> - I see a common pattern in definitions of using `dip` instead of
> `swap`. Is there some special reason for that? Is it more performant? I
> know the words aren't interchangeable but e.g. `with-directory-files`
> has `[ [ "" directory-files ] ] dip` which as far as I can tell is
> equivalent to `[ "" directory-files ] swap`. I saw this pattern in more
> definitions.
>
I guess it's a matter of personal style. I would argue that the 'meaning'
of swap ( x y -- y x ) is that there's the same importance on pushing y
down the stack than on puling x up the stack. Whereas [ y ] dip would focus
more putting y down the stack.

Regarding the performance, you can often see for yourself using the
optimized. word of compiler.tree.debugger. You could even install libudis
and use the disassemble word of tools.disassembler.
I would be surprised if the performance of swap vs dip mattered in a real
application.


>
> - is there any sequence generator/iterator vocabulary? Something that
> gives or computes values on demand. One can find it in many languages
> with a bit different flavor, e.g. Scheme, Rust, Python. I saw that there
> is lists.lazy which can serve a similar purpose but is a bit more heavy
> weight in some cases. Maybe this isn't needed in factor at all and you
> use a different pattern to solve a similar problem?
>
I've seen discussions on this mailing list about the extra/cursors
vocabulary about that. I've never used it though. For example Joe talked
about it in 2009: https://sourceforge.net/p/factor/mailman/message/23878123

Cheers,
Jon
--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Factor-talk mailing list
Factor-talk@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/factor-talk