Re: help with global replace in text file/list of bytes

[dean]

I've converted all of my filters to use in/out and have made the above
virtually identical to that Rosetta Code example and the result is
now.virtually instantaneousjust for the cost of a couple of
temporary files. The difference is remarkable and again thank you both for
steering me in this direction.

(de lchg (From To Ipth Opth)
   (out Opth
  (in Ipth
 (while (echo From)
(prin To)

On 21 February 2017 at 15:20, dean  wrote:

> PS ..yes that Rosetta Code example is close to what I'm after.
>
> On 21 February 2017 at 15:19, dean  wrote:
>
>> Hi Andreas
>> >Do you really need to load all the stuff into RAM?
>> No...I was originally using in/out
>> and will go back to trying that for each filter.
>>
>> re the improvements...yes I understand all all of those...thank you.
>> >exactly the same structure as in ram.
>> Amazing and I'll bear that in mind.
>>
>> >I hope this helps...
>> Yes it does and thank you very much
>>
>> Hi Joe
>> >1. Your picolisp code is becoming easier to read.
>> That's very kind...thank you...I am trying :)
>>
>> >3. Why do you need to work with bytes vs chars?
>> Good question...Initially I have some >130 asci characters.
>> Reading them as such is problematic but viewing them as bytes
>> and changing those >asci 130 solves the problem.
>> As the data is in bytes after such conversion I thought it might be
>> more efficient to keep them that way but after
>> your question,
>> Rosetta Code example
>> and Andreas' advice re confining things to in/out
>> maybe not.
>> I'll have a go trying to incorporate the above advice and example
>> and report back. It might take me a while.
>>
>> Thank you both very much for your help.
>> Best Regards
>> Dean
>>
>> On 21 February 2017 at 13:14, Joe Bogner  wrote:
>>
>>> After trying to figure it out myself for a few minutes, I remembered to
>>> check rosettacode (wonderful resource). This is probably close to what you
>>> need: http://rosettacode.org/wiki/Globally_replace_text_in_s
>>> everal_files#PicoLisp
>>>
>>> On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner  wrote:
>>>
 Hi dean,

 I experimented with this problem for a few minutes and didn't come up
 with anything worth posting. A few comments though:

 1. Your picolisp code is becoming easier to read. Nice work!
 2. My initial thought was to split the input into words and replace
 sublists, however it looks like you don't have a word delimiter (typically
 a space)...Since you need to be able to substitute "fl ow" with "flow". As
 a result, the best I came up with is something similar (looping through
 characters and testing the replacement)
 3. Why do you need to work with bytes vs chars?  (mapcar char (chop
 Sfrom)) ?



 On Tue, Feb 21, 2017 at 3:37 AM, dean  wrote:

> I need to globally replace certain words in a text file and because
> I need to process it a byte at a time initially...I'm inputting
> processed list of bytes into the global replace function "lchg"
> (and others) like this.
>
> (lbytes_to_fl Cleaned_txt_pth
>(lchg "fl ow" "flow"
>(fltr2
>(fltr1
> (fl_to_lbytes Txt_pth)
>
> The other filters seem ok but this one is slow (most likely my
> algorithm/general approach :)) and any help to
> speed things up would be much appreciated.
>
> (de lchg (Sfrom Sto Lbytes)
>(make
>   (let
>  (X 0
> B NIL
> Lfrom (mapcar char (chop Sfrom))
> Lto (mapcar char (chop Sto))
> First_from_ch (car Lfrom)
> Len_from-1 (- (length Lfrom) 1)
> Len_lbytes (length Lbytes) )
>  (until (<= (length Lbytes) X)
> (inc 'X)
> (setq B (get Lbytes X))
> (if (= B First_from_ch)
>(prog
>   (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom)
>  (prog
> (for MatchB Lto
>(link MatchB) )
> (inc 'X Len_from-1) )
>  (link B) ) )
>(link B) ) ) ) ) )
>
>(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99
>
> Here's "lchg" in action...
>
> : (setq L (chop "ab fl ow flow fl ow yz"))
> -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" "
> " "o" "w" " " "y" "z")
> : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L
> -> "ab flow flow flow yz"
>
>

>>>
>>
>

Re: help with global replace in text file/list of bytes

[dean]

PS ..yes that Rosetta Code example is close to what I'm after.

On 21 February 2017 at 15:19, dean  wrote:

> Hi Andreas
> >Do you really need to load all the stuff into RAM?
> No...I was originally using in/out
> and will go back to trying that for each filter.
>
> re the improvements...yes I understand all all of those...thank you.
> >exactly the same structure as in ram.
> Amazing and I'll bear that in mind.
>
> >I hope this helps...
> Yes it does and thank you very much
>
> Hi Joe
> >1. Your picolisp code is becoming easier to read.
> That's very kind...thank you...I am trying :)
>
> >3. Why do you need to work with bytes vs chars?
> Good question...Initially I have some >130 asci characters.
> Reading them as such is problematic but viewing them as bytes
> and changing those >asci 130 solves the problem.
> As the data is in bytes after such conversion I thought it might be
> more efficient to keep them that way but after
> your question,
> Rosetta Code example
> and Andreas' advice re confining things to in/out
> maybe not.
> I'll have a go trying to incorporate the above advice and example
> and report back. It might take me a while.
>
> Thank you both very much for your help.
> Best Regards
> Dean
>
> On 21 February 2017 at 13:14, Joe Bogner  wrote:
>
>> After trying to figure it out myself for a few minutes, I remembered to
>> check rosettacode (wonderful resource). This is probably close to what you
>> need: http://rosettacode.org/wiki/Globally_replace_text_in_s
>> everal_files#PicoLisp
>>
>> On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner  wrote:
>>
>>> Hi dean,
>>>
>>> I experimented with this problem for a few minutes and didn't come up
>>> with anything worth posting. A few comments though:
>>>
>>> 1. Your picolisp code is becoming easier to read. Nice work!
>>> 2. My initial thought was to split the input into words and replace
>>> sublists, however it looks like you don't have a word delimiter (typically
>>> a space)...Since you need to be able to substitute "fl ow" with "flow". As
>>> a result, the best I came up with is something similar (looping through
>>> characters and testing the replacement)
>>> 3. Why do you need to work with bytes vs chars?  (mapcar char (chop
>>> Sfrom)) ?
>>>
>>>
>>>
>>> On Tue, Feb 21, 2017 at 3:37 AM, dean  wrote:
>>>
 I need to globally replace certain words in a text file and because
 I need to process it a byte at a time initially...I'm inputting
 processed list of bytes into the global replace function "lchg"
 (and others) like this.

 (lbytes_to_fl Cleaned_txt_pth
(lchg "fl ow" "flow"
(fltr2
(fltr1
 (fl_to_lbytes Txt_pth)

 The other filters seem ok but this one is slow (most likely my
 algorithm/general approach :)) and any help to
 speed things up would be much appreciated.

 (de lchg (Sfrom Sto Lbytes)
(make
   (let
  (X 0
 B NIL
 Lfrom (mapcar char (chop Sfrom))
 Lto (mapcar char (chop Sto))
 First_from_ch (car Lfrom)
 Len_from-1 (- (length Lfrom) 1)
 Len_lbytes (length Lbytes) )
  (until (<= (length Lbytes) X)
 (inc 'X)
 (setq B (get Lbytes X))
 (if (= B First_from_ch)
(prog
   (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom)
  (prog
 (for MatchB Lto
(link MatchB) )
 (inc 'X Len_from-1) )
  (link B) ) )
(link B) ) ) ) ) )

(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99

 Here's "lchg" in action...

 : (setq L (chop "ab fl ow flow fl ow yz"))
 -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " "
 "o" "w" " " "y" "z")
 : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L
 -> "ab flow flow flow yz"


>>>
>>
>

Re: help with global replace in text file/list of bytes

[dean]

Hi Andreas
>Do you really need to load all the stuff into RAM?
No...I was originally using in/out
and will go back to trying that for each filter.

re the improvements...yes I understand all all of those...thank you.
>exactly the same structure as in ram.
Amazing and I'll bear that in mind.

>I hope this helps...
Yes it does and thank you very much

Hi Joe
>1. Your picolisp code is becoming easier to read.
That's very kind...thank you...I am trying :)

>3. Why do you need to work with bytes vs chars?
Good question...Initially I have some >130 asci characters.
Reading them as such is problematic but viewing them as bytes
and changing those >asci 130 solves the problem.
As the data is in bytes after such conversion I thought it might be
more efficient to keep them that way but after
your question,
Rosetta Code example
and Andreas' advice re confining things to in/out
...maybe not.
I'll have a go trying to incorporate the above advice and example
and report back. It might take me a while.

Thank you both very much for your help.
Best Regards
Dean

On 21 February 2017 at 13:14, Joe Bogner  wrote:

> After trying to figure it out myself for a few minutes, I remembered to
> check rosettacode (wonderful resource). This is probably close to what you
> need: http://rosettacode.org/wiki/Globally_replace_text_in_
> several_files#PicoLisp
>
> On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner  wrote:
>
>> Hi dean,
>>
>> I experimented with this problem for a few minutes and didn't come up
>> with anything worth posting. A few comments though:
>>
>> 1. Your picolisp code is becoming easier to read. Nice work!
>> 2. My initial thought was to split the input into words and replace
>> sublists, however it looks like you don't have a word delimiter (typically
>> a space)...Since you need to be able to substitute "fl ow" with "flow". As
>> a result, the best I came up with is something similar (looping through
>> characters and testing the replacement)
>> 3. Why do you need to work with bytes vs chars?  (mapcar char (chop
>> Sfrom)) ?
>>
>>
>>
>> On Tue, Feb 21, 2017 at 3:37 AM, dean  wrote:
>>
>>> I need to globally replace certain words in a text file and because
>>> I need to process it a byte at a time initially...I'm inputting
>>> processed list of bytes into the global replace function "lchg"
>>> (and others) like this.
>>>
>>> (lbytes_to_fl Cleaned_txt_pth
>>>(lchg "fl ow" "flow"
>>>(fltr2
>>>(fltr1
>>> (fl_to_lbytes Txt_pth)
>>>
>>> The other filters seem ok but this one is slow (most likely my
>>> algorithm/general approach :)) and any help to
>>> speed things up would be much appreciated.
>>>
>>> (de lchg (Sfrom Sto Lbytes)
>>>(make
>>>   (let
>>>  (X 0
>>> B NIL
>>> Lfrom (mapcar char (chop Sfrom))
>>> Lto (mapcar char (chop Sto))
>>> First_from_ch (car Lfrom)
>>> Len_from-1 (- (length Lfrom) 1)
>>> Len_lbytes (length Lbytes) )
>>>  (until (<= (length Lbytes) X)
>>> (inc 'X)
>>> (setq B (get Lbytes X))
>>> (if (= B First_from_ch)
>>>(prog
>>>   (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom)
>>>  (prog
>>> (for MatchB Lto
>>>(link MatchB) )
>>> (inc 'X Len_from-1) )
>>>  (link B) ) )
>>>(link B) ) ) ) ) )
>>>
>>>(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99
>>>
>>> Here's "lchg" in action...
>>>
>>> : (setq L (chop "ab fl ow flow fl ow yz"))
>>> -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " "
>>> "o" "w" " " "y" "z")
>>> : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L
>>> -> "ab flow flow flow yz"
>>>
>>>
>>
>

Re: help with global replace in text file/list of bytes

[Joe Bogner]

After trying to figure it out myself for a few minutes, I remembered to
check rosettacode (wonderful resource). This is probably close to what you
need:
http://rosettacode.org/wiki/Globally_replace_text_in_several_files#PicoLisp

On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner  wrote:

> Hi dean,
>
> I experimented with this problem for a few minutes and didn't come up with
> anything worth posting. A few comments though:
>
> 1. Your picolisp code is becoming easier to read. Nice work!
> 2. My initial thought was to split the input into words and replace
> sublists, however it looks like you don't have a word delimiter (typically
> a space)...Since you need to be able to substitute "fl ow" with "flow". As
> a result, the best I came up with is something similar (looping through
> characters and testing the replacement)
> 3. Why do you need to work with bytes vs chars?  (mapcar char (chop
> Sfrom)) ?
>
>
>
> On Tue, Feb 21, 2017 at 3:37 AM, dean  wrote:
>
>> I need to globally replace certain words in a text file and because
>> I need to process it a byte at a time initially...I'm inputting
>> processed list of bytes into the global replace function "lchg"
>> (and others) like this.
>>
>> (lbytes_to_fl Cleaned_txt_pth
>>(lchg "fl ow" "flow"
>>(fltr2
>>(fltr1
>> (fl_to_lbytes Txt_pth)
>>
>> The other filters seem ok but this one is slow (most likely my
>> algorithm/general approach :)) and any help to
>> speed things up would be much appreciated.
>>
>> (de lchg (Sfrom Sto Lbytes)
>>(make
>>   (let
>>  (X 0
>> B NIL
>> Lfrom (mapcar char (chop Sfrom))
>> Lto (mapcar char (chop Sto))
>> First_from_ch (car Lfrom)
>> Len_from-1 (- (length Lfrom) 1)
>> Len_lbytes (length Lbytes) )
>>  (until (<= (length Lbytes) X)
>> (inc 'X)
>> (setq B (get Lbytes X))
>> (if (= B First_from_ch)
>>(prog
>>   (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom)
>>  (prog
>> (for MatchB Lto
>>(link MatchB) )
>> (inc 'X Len_from-1) )
>>  (link B) ) )
>>(link B) ) ) ) ) )
>>
>>(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99
>>
>> Here's "lchg" in action...
>>
>> : (setq L (chop "ab fl ow flow fl ow yz"))
>> -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " "
>> "o" "w" " " "y" "z")
>> : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L
>> -> "ab flow flow flow yz"
>>
>>
>

Re: help with global replace in text file/list of bytes

[Joe Bogner]

Hi dean,

I experimented with this problem for a few minutes and didn't come up with
anything worth posting. A few comments though:

1. Your picolisp code is becoming easier to read. Nice work!
2. My initial thought was to split the input into words and replace
sublists, however it looks like you don't have a word delimiter (typically
a space)...Since you need to be able to substitute "fl ow" with "flow". As
a result, the best I came up with is something similar (looping through
characters and testing the replacement)
3. Why do you need to work with bytes vs chars?  (mapcar char (chop Sfrom))
?



On Tue, Feb 21, 2017 at 3:37 AM, dean  wrote:

> I need to globally replace certain words in a text file and because
> I need to process it a byte at a time initially...I'm inputting
> processed list of bytes into the global replace function "lchg"
> (and others) like this.
>
> (lbytes_to_fl Cleaned_txt_pth
>(lchg "fl ow" "flow"
>(fltr2
>(fltr1
> (fl_to_lbytes Txt_pth)
>
> The other filters seem ok but this one is slow (most likely my
> algorithm/general approach :)) and any help to
> speed things up would be much appreciated.
>
> (de lchg (Sfrom Sto Lbytes)
>(make
>   (let
>  (X 0
> B NIL
> Lfrom (mapcar char (chop Sfrom))
> Lto (mapcar char (chop Sto))
> First_from_ch (car Lfrom)
> Len_from-1 (- (length Lfrom) 1)
> Len_lbytes (length Lbytes) )
>  (until (<= (length Lbytes) X)
> (inc 'X)
> (setq B (get Lbytes X))
> (if (= B First_from_ch)
>(prog
>   (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom)
>  (prog
> (for MatchB Lto
>(link MatchB) )
> (inc 'X Len_from-1) )
>  (link B) ) )
>(link B) ) ) ) ) )
>
>(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99
>
> Here's "lchg" in action...
>
> : (setq L (chop "ab fl ow flow fl ow yz"))
> -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " "
> "o" "w" " " "y" "z")
> : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L
> -> "ab flow flow flow yz"
>
>

RE: help with global replace in text file/list of bytes

[andreas]

Hi dean

Do you really need to load all the stuff into RAM?
Working in stream, e.g. with (in) (out) and (char) (called without arguments) 
or (rd) (binary read) needs slightly different software design, but would most 
likely be faster.

Also, (chop) is a rather expensive function (because it has to reserve new 
memory), probably you can easily improve performance by chopping all input once 
during initial reading, and then store the content as lists instead of strings 
which you chop in every filter function.
Maybe you can identify other areas where you could keep some stuff in variables 
instead of doing the same thing multiple times.

Then make sure that you don't have any code multiple times lying around, 
refactor common steps into functions.
In picolisp the source code is exactly the same structure as the RAM 
representation and execution flow,  it follows that shorter code usually 
results in better performance.

I hope this helps, good luck and have fun!

Regards,
beneroth

help with global replace in text file/list of bytes

[dean]

I need to globally replace certain words in a text file and because
I need to process it a byte at a time initially...I'm inputting
processed list of bytes into the global replace function "lchg"
(and others) like this.

(lbytes_to_fl Cleaned_txt_pth
   (lchg "fl ow" "flow"
   (fltr2
   (fltr1
(fl_to_lbytes Txt_pth)

The other filters seem ok but this one is slow (most likely my
algorithm/general approach :)) and any help to
speed things up would be much appreciated.

(de lchg (Sfrom Sto Lbytes)
   (make
  (let
 (X 0
B NIL
Lfrom (mapcar char (chop Sfrom))
Lto (mapcar char (chop Sto))
First_from_ch (car Lfrom)
Len_from-1 (- (length Lfrom) 1)
Len_lbytes (length Lbytes) )
 (until (<= (length Lbytes) X)
(inc 'X)
(setq B (get Lbytes X))
(if (= B First_from_ch)
   (prog
  (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom)
 (prog
(for MatchB Lto
   (link MatchB) )
(inc 'X Len_from-1) )
 (link B) ) )
   (link B) ) ) ) ) )

   (de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99

Here's "lchg" in action...

: (setq L (chop "ab fl ow flow fl ow yz"))
-> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " " "o"
"w" " " "y" "z")
: (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L
-> "ab flow flow flow yz"