Re: help with global replace in text file/list of bytes
I've converted all of my filters to use in/out and have made the above virtually identical to that Rosetta Code example and the result is now.virtually instantaneousjust for the cost of a couple of temporary files. The difference is remarkable and again thank you both for steering me in this direction. (de lchg (From To Ipth Opth) (out Opth (in Ipth (while (echo From) (prin To) On 21 February 2017 at 15:20, dean wrote: > PS ..yes that Rosetta Code example is close to what I'm after. > > On 21 February 2017 at 15:19, dean wrote: > >> Hi Andreas >> >Do you really need to load all the stuff into RAM? >> No...I was originally using in/out >> and will go back to trying that for each filter. >> >> re the improvements...yes I understand all all of those...thank you. >> >exactly the same structure as in ram. >> Amazing and I'll bear that in mind. >> >> >I hope this helps... >> Yes it does and thank you very much >> >> Hi Joe >> >1. Your picolisp code is becoming easier to read. >> That's very kind...thank you...I am trying :) >> >> >3. Why do you need to work with bytes vs chars? >> Good question...Initially I have some >130 asci characters. >> Reading them as such is problematic but viewing them as bytes >> and changing those >asci 130 solves the problem. >> As the data is in bytes after such conversion I thought it might be >> more efficient to keep them that way but after >> your question, >> Rosetta Code example >> and Andreas' advice re confining things to in/out >> maybe not. >> I'll have a go trying to incorporate the above advice and example >> and report back. It might take me a while. >> >> Thank you both very much for your help. >> Best Regards >> Dean >> >> On 21 February 2017 at 13:14, Joe Bogner wrote: >> >>> After trying to figure it out myself for a few minutes, I remembered to >>> check rosettacode (wonderful resource). This is probably close to what you >>> need: http://rosettacode.org/wiki/Globally_replace_text_in_s >>> everal_files#PicoLisp >>> >>> On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner wrote: >>> Hi dean, I experimented with this problem for a few minutes and didn't come up with anything worth posting. A few comments though: 1. Your picolisp code is becoming easier to read. Nice work! 2. My initial thought was to split the input into words and replace sublists, however it looks like you don't have a word delimiter (typically a space)...Since you need to be able to substitute "fl ow" with "flow". As a result, the best I came up with is something similar (looping through characters and testing the replacement) 3. Why do you need to work with bytes vs chars? (mapcar char (chop Sfrom)) ? On Tue, Feb 21, 2017 at 3:37 AM, dean wrote: > I need to globally replace certain words in a text file and because > I need to process it a byte at a time initially...I'm inputting > processed list of bytes into the global replace function "lchg" > (and others) like this. > > (lbytes_to_fl Cleaned_txt_pth >(lchg "fl ow" "flow" >(fltr2 >(fltr1 > (fl_to_lbytes Txt_pth) > > The other filters seem ok but this one is slow (most likely my > algorithm/general approach :)) and any help to > speed things up would be much appreciated. > > (de lchg (Sfrom Sto Lbytes) >(make > (let > (X 0 > B NIL > Lfrom (mapcar char (chop Sfrom)) > Lto (mapcar char (chop Sto)) > First_from_ch (car Lfrom) > Len_from-1 (- (length Lfrom) 1) > Len_lbytes (length Lbytes) ) > (until (<= (length Lbytes) X) > (inc 'X) > (setq B (get Lbytes X)) > (if (= B First_from_ch) >(prog > (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom) > (prog > (for MatchB Lto >(link MatchB) ) > (inc 'X Len_from-1) ) > (link B) ) ) >(link B) ) ) ) ) ) > >(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99 > > Here's "lchg" in action... > > : (setq L (chop "ab fl ow flow fl ow yz")) > -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " > " "o" "w" " " "y" "z") > : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L > -> "ab flow flow flow yz" > > >>> >> >
Re: help with global replace in text file/list of bytes
PS ..yes that Rosetta Code example is close to what I'm after. On 21 February 2017 at 15:19, dean wrote: > Hi Andreas > >Do you really need to load all the stuff into RAM? > No...I was originally using in/out > and will go back to trying that for each filter. > > re the improvements...yes I understand all all of those...thank you. > >exactly the same structure as in ram. > Amazing and I'll bear that in mind. > > >I hope this helps... > Yes it does and thank you very much > > Hi Joe > >1. Your picolisp code is becoming easier to read. > That's very kind...thank you...I am trying :) > > >3. Why do you need to work with bytes vs chars? > Good question...Initially I have some >130 asci characters. > Reading them as such is problematic but viewing them as bytes > and changing those >asci 130 solves the problem. > As the data is in bytes after such conversion I thought it might be > more efficient to keep them that way but after > your question, > Rosetta Code example > and Andreas' advice re confining things to in/out > maybe not. > I'll have a go trying to incorporate the above advice and example > and report back. It might take me a while. > > Thank you both very much for your help. > Best Regards > Dean > > On 21 February 2017 at 13:14, Joe Bogner wrote: > >> After trying to figure it out myself for a few minutes, I remembered to >> check rosettacode (wonderful resource). This is probably close to what you >> need: http://rosettacode.org/wiki/Globally_replace_text_in_s >> everal_files#PicoLisp >> >> On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner wrote: >> >>> Hi dean, >>> >>> I experimented with this problem for a few minutes and didn't come up >>> with anything worth posting. A few comments though: >>> >>> 1. Your picolisp code is becoming easier to read. Nice work! >>> 2. My initial thought was to split the input into words and replace >>> sublists, however it looks like you don't have a word delimiter (typically >>> a space)...Since you need to be able to substitute "fl ow" with "flow". As >>> a result, the best I came up with is something similar (looping through >>> characters and testing the replacement) >>> 3. Why do you need to work with bytes vs chars? (mapcar char (chop >>> Sfrom)) ? >>> >>> >>> >>> On Tue, Feb 21, 2017 at 3:37 AM, dean wrote: >>> I need to globally replace certain words in a text file and because I need to process it a byte at a time initially...I'm inputting processed list of bytes into the global replace function "lchg" (and others) like this. (lbytes_to_fl Cleaned_txt_pth (lchg "fl ow" "flow" (fltr2 (fltr1 (fl_to_lbytes Txt_pth) The other filters seem ok but this one is slow (most likely my algorithm/general approach :)) and any help to speed things up would be much appreciated. (de lchg (Sfrom Sto Lbytes) (make (let (X 0 B NIL Lfrom (mapcar char (chop Sfrom)) Lto (mapcar char (chop Sto)) First_from_ch (car Lfrom) Len_from-1 (- (length Lfrom) 1) Len_lbytes (length Lbytes) ) (until (<= (length Lbytes) X) (inc 'X) (setq B (get Lbytes X)) (if (= B First_from_ch) (prog (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom) (prog (for MatchB Lto (link MatchB) ) (inc 'X Len_from-1) ) (link B) ) ) (link B) ) ) ) ) ) (de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99 Here's "lchg" in action... : (setq L (chop "ab fl ow flow fl ow yz")) -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " " "o" "w" " " "y" "z") : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L -> "ab flow flow flow yz" >>> >> >
Re: help with global replace in text file/list of bytes
Hi Andreas >Do you really need to load all the stuff into RAM? No...I was originally using in/out and will go back to trying that for each filter. re the improvements...yes I understand all all of those...thank you. >exactly the same structure as in ram. Amazing and I'll bear that in mind. >I hope this helps... Yes it does and thank you very much Hi Joe >1. Your picolisp code is becoming easier to read. That's very kind...thank you...I am trying :) >3. Why do you need to work with bytes vs chars? Good question...Initially I have some >130 asci characters. Reading them as such is problematic but viewing them as bytes and changing those >asci 130 solves the problem. As the data is in bytes after such conversion I thought it might be more efficient to keep them that way but after your question, Rosetta Code example and Andreas' advice re confining things to in/out ...maybe not. I'll have a go trying to incorporate the above advice and example and report back. It might take me a while. Thank you both very much for your help. Best Regards Dean On 21 February 2017 at 13:14, Joe Bogner wrote: > After trying to figure it out myself for a few minutes, I remembered to > check rosettacode (wonderful resource). This is probably close to what you > need: http://rosettacode.org/wiki/Globally_replace_text_in_ > several_files#PicoLisp > > On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner wrote: > >> Hi dean, >> >> I experimented with this problem for a few minutes and didn't come up >> with anything worth posting. A few comments though: >> >> 1. Your picolisp code is becoming easier to read. Nice work! >> 2. My initial thought was to split the input into words and replace >> sublists, however it looks like you don't have a word delimiter (typically >> a space)...Since you need to be able to substitute "fl ow" with "flow". As >> a result, the best I came up with is something similar (looping through >> characters and testing the replacement) >> 3. Why do you need to work with bytes vs chars? (mapcar char (chop >> Sfrom)) ? >> >> >> >> On Tue, Feb 21, 2017 at 3:37 AM, dean wrote: >> >>> I need to globally replace certain words in a text file and because >>> I need to process it a byte at a time initially...I'm inputting >>> processed list of bytes into the global replace function "lchg" >>> (and others) like this. >>> >>> (lbytes_to_fl Cleaned_txt_pth >>>(lchg "fl ow" "flow" >>>(fltr2 >>>(fltr1 >>> (fl_to_lbytes Txt_pth) >>> >>> The other filters seem ok but this one is slow (most likely my >>> algorithm/general approach :)) and any help to >>> speed things up would be much appreciated. >>> >>> (de lchg (Sfrom Sto Lbytes) >>>(make >>> (let >>> (X 0 >>> B NIL >>> Lfrom (mapcar char (chop Sfrom)) >>> Lto (mapcar char (chop Sto)) >>> First_from_ch (car Lfrom) >>> Len_from-1 (- (length Lfrom) 1) >>> Len_lbytes (length Lbytes) ) >>> (until (<= (length Lbytes) X) >>> (inc 'X) >>> (setq B (get Lbytes X)) >>> (if (= B First_from_ch) >>>(prog >>> (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom) >>> (prog >>> (for MatchB Lto >>>(link MatchB) ) >>> (inc 'X Len_from-1) ) >>> (link B) ) ) >>>(link B) ) ) ) ) ) >>> >>>(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99 >>> >>> Here's "lchg" in action... >>> >>> : (setq L (chop "ab fl ow flow fl ow yz")) >>> -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " " >>> "o" "w" " " "y" "z") >>> : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L >>> -> "ab flow flow flow yz" >>> >>> >> >
Re: help with global replace in text file/list of bytes
After trying to figure it out myself for a few minutes, I remembered to check rosettacode (wonderful resource). This is probably close to what you need: http://rosettacode.org/wiki/Globally_replace_text_in_several_files#PicoLisp On Tue, Feb 21, 2017 at 8:08 AM, Joe Bogner wrote: > Hi dean, > > I experimented with this problem for a few minutes and didn't come up with > anything worth posting. A few comments though: > > 1. Your picolisp code is becoming easier to read. Nice work! > 2. My initial thought was to split the input into words and replace > sublists, however it looks like you don't have a word delimiter (typically > a space)...Since you need to be able to substitute "fl ow" with "flow". As > a result, the best I came up with is something similar (looping through > characters and testing the replacement) > 3. Why do you need to work with bytes vs chars? (mapcar char (chop > Sfrom)) ? > > > > On Tue, Feb 21, 2017 at 3:37 AM, dean wrote: > >> I need to globally replace certain words in a text file and because >> I need to process it a byte at a time initially...I'm inputting >> processed list of bytes into the global replace function "lchg" >> (and others) like this. >> >> (lbytes_to_fl Cleaned_txt_pth >>(lchg "fl ow" "flow" >>(fltr2 >>(fltr1 >> (fl_to_lbytes Txt_pth) >> >> The other filters seem ok but this one is slow (most likely my >> algorithm/general approach :)) and any help to >> speed things up would be much appreciated. >> >> (de lchg (Sfrom Sto Lbytes) >>(make >> (let >> (X 0 >> B NIL >> Lfrom (mapcar char (chop Sfrom)) >> Lto (mapcar char (chop Sto)) >> First_from_ch (car Lfrom) >> Len_from-1 (- (length Lfrom) 1) >> Len_lbytes (length Lbytes) ) >> (until (<= (length Lbytes) X) >> (inc 'X) >> (setq B (get Lbytes X)) >> (if (= B First_from_ch) >>(prog >> (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom) >> (prog >> (for MatchB Lto >>(link MatchB) ) >> (inc 'X Len_from-1) ) >> (link B) ) ) >>(link B) ) ) ) ) ) >> >>(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99 >> >> Here's "lchg" in action... >> >> : (setq L (chop "ab fl ow flow fl ow yz")) >> -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " " >> "o" "w" " " "y" "z") >> : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L >> -> "ab flow flow flow yz" >> >> >
Re: help with global replace in text file/list of bytes
Hi dean, I experimented with this problem for a few minutes and didn't come up with anything worth posting. A few comments though: 1. Your picolisp code is becoming easier to read. Nice work! 2. My initial thought was to split the input into words and replace sublists, however it looks like you don't have a word delimiter (typically a space)...Since you need to be able to substitute "fl ow" with "flow". As a result, the best I came up with is something similar (looping through characters and testing the replacement) 3. Why do you need to work with bytes vs chars? (mapcar char (chop Sfrom)) ? On Tue, Feb 21, 2017 at 3:37 AM, dean wrote: > I need to globally replace certain words in a text file and because > I need to process it a byte at a time initially...I'm inputting > processed list of bytes into the global replace function "lchg" > (and others) like this. > > (lbytes_to_fl Cleaned_txt_pth >(lchg "fl ow" "flow" >(fltr2 >(fltr1 > (fl_to_lbytes Txt_pth) > > The other filters seem ok but this one is slow (most likely my > algorithm/general approach :)) and any help to > speed things up would be much appreciated. > > (de lchg (Sfrom Sto Lbytes) >(make > (let > (X 0 > B NIL > Lfrom (mapcar char (chop Sfrom)) > Lto (mapcar char (chop Sto)) > First_from_ch (car Lfrom) > Len_from-1 (- (length Lfrom) 1) > Len_lbytes (length Lbytes) ) > (until (<= (length Lbytes) X) > (inc 'X) > (setq B (get Lbytes X)) > (if (= B First_from_ch) >(prog > (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom) > (prog > (for MatchB Lto >(link MatchB) ) > (inc 'X Len_from-1) ) > (link B) ) ) >(link B) ) ) ) ) ) > >(de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99 > > Here's "lchg" in action... > > : (setq L (chop "ab fl ow flow fl ow yz")) > -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " " > "o" "w" " " "y" "z") > : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L > -> "ab flow flow flow yz" > >
RE: help with global replace in text file/list of bytes
Hi dean Do you really need to load all the stuff into RAM? Working in stream, e.g. with (in) (out) and (char) (called without arguments) or (rd) (binary read) needs slightly different software design, but would most likely be faster. Also, (chop) is a rather expensive function (because it has to reserve new memory), probably you can easily improve performance by chopping all input once during initial reading, and then store the content as lists instead of strings which you chop in every filter function. Maybe you can identify other areas where you could keep some stuff in variables instead of doing the same thing multiple times. Then make sure that you don't have any code multiple times lying around, refactor common steps into functions. In picolisp the source code is exactly the same structure as the RAM representation and execution flow, it follows that shorter code usually results in better performance. I hope this helps, good luck and have fun! Regards, beneroth
help with global replace in text file/list of bytes
I need to globally replace certain words in a text file and because I need to process it a byte at a time initially...I'm inputting processed list of bytes into the global replace function "lchg" (and others) like this. (lbytes_to_fl Cleaned_txt_pth (lchg "fl ow" "flow" (fltr2 (fltr1 (fl_to_lbytes Txt_pth) The other filters seem ok but this one is slow (most likely my algorithm/general approach :)) and any help to speed things up would be much appreciated. (de lchg (Sfrom Sto Lbytes) (make (let (X 0 B NIL Lfrom (mapcar char (chop Sfrom)) Lto (mapcar char (chop Sto)) First_from_ch (car Lfrom) Len_from-1 (- (length Lfrom) 1) Len_lbytes (length Lbytes) ) (until (<= (length Lbytes) X) (inc 'X) (setq B (get Lbytes X)) (if (= B First_from_ch) (prog (if (= (slice Lbytes X (+ X Len_from-1)) Lfrom) (prog (for MatchB Lto (link MatchB) ) (inc 'X Len_from-1) ) (link B) ) ) (link B) ) ) ) ) ) (de slice (Lst I K) (head (inc (- K I)) (nth Lst I)) ) #99 Here's "lchg" in action... : (setq L (chop "ab fl ow flow fl ow yz")) -> ("a" "b" " " "f" "l" " " "o" "w" " " "f" "l" "o" "w" " " "f" "l" " " "o" "w" " " "y" "z") : (pack (mapcar char (lchg "fl ow" "flow" (mapcar char L -> "ab flow flow flow yz"