On Monday, January 18, 2016 at 10:23:56 AM UTC-5, gustavo wrote: > I have a few minor stile comments: > > *** I'd replace > (define backslash 92) > with > (define backslash (char->integer #\\)) > to improve legibility. > And do the same replacement for other magic numbers. After > optimization, both versions are identical. > > [The only site where this would cause a difference is in code like > (for ([i (in-range 92)]) ...) > but you are not using something like this.] > > > *** In few spots, you use `null` as "do nothing in this case", for > example in line 38. I think it's more idiomatic to use `(void)`. > > *** In line 328, I it possible to do this? Replace > > ;;[[[ > ;; 5. Remove all occurrences of AEIOU, except first letter > (define len2 (bytes-length str4)) > (define str5(bytes-append > (subbytes str4 0 1) > (bytes-delete-pred > (if (> len2 1) (subbytes str4 1 len2) #"") > (λ (b) ...)))) > ;; 6. If first symbol is a digit replace it with letter saved on step 1. > (when (byte-numeric? (bytes-ref str5 0)) > (bytes-set! str5 0 first_letter)) > ;;;]]] > > with this > > ;;[[[ > ;; 5. Remove all occurrences of AEIOU, except first letter > (define len2 (bytes-length str4)) > (define str5(bytes-append > #"?" > (bytes-delete-pred > (if (> len2 1) (subbytes str4 1 len2) #"") > (λ (b) ...)))) > ;; 6. Replace firt symbol with letter saved on step 1. > (bytes-set! str5 0 first_letter) > ;;;]]] > > or this: > > ;;[[[ > ;; 5. Remove all occurrences > (define len2 (bytes-length str4)) > (when (>= len2 1) > (bytes-set! str4 0 (char->integer #\?)) > (define str5 > (bytes-delete-pred > str4 > (λ (b) ...)))) > ;; 6. Replace firt symbol with letter saved on step 1. > (when (>= len2 1) > (bytes-set! str5 0 first_letter)) > ;;;]]] > > I hope I'm not missing a corner case. I estimate that with the 200K > fields this will reduce the time from 2.5s to 2.4s. > > > *** I tried a few more changes that avoid allocating intermediate > bytes, but they are more complex and with 200K rows I estimate that > the difference in run time would be only 0.02s. If I find something > bigger, I'll write again. > > Gustavo > > PS: How many "JOHN"s do you have? Have you considered caching the > soundex of the names. But I'm not sure if this would be faster. > > On Sun, Jan 17, 2016 at 10:13 PM, Brian Adkins wrote: > > On Sunday, January 17, 2016 at 2:54:39 PM UTC-5, Brian Adkins wrote: > >> On Sunday, January 17, 2016 at 2:50:19 PM UTC-5, Brian Adkins wrote: > >> > > >> > With built-in string-trim, the lowest of three runs was 10293. Using > >> > your string-trim the lowest of three runs was 7618, so it reduced the > >> > runtime by 26%. > >> > >> Although, I probably should've mentioned that I'm not particularly > >> interested in unsafe optimizations. I already have a very fast C program > >> if I'm willing to risk unsafe behavior, so for Racket, I'd like to retain > >> safety. > >> > >> Having said that, I'm pretty sure a combination of using Byte Strings and > >> manually optimizing string-trim & string-replace (or skipping them in some > >> cases) will get under the Ruby time. > > > > Yay - success! :) > > > > I changed all strings to byte strings while leaving the style of the code > > very similar. It made a significant difference. Of course, I also gained > > the benefit if handwritten bytes-split, bytes-replace, bytes-delete, > > bytes-trim, etc. which were narrowly defined just for this app. > > > > Timings on a 200K line file are now: > > > > Ruby = 7.53s > > Racket = 2.52s (was 10.3s) > > > > The string version of the Racket program was over 4x slower. I'm quite > > satisfied with being 3x faster than Ruby with a similar coding style given > > this is really in Ruby's sweet spot i.e. text munging. > > > > New code is here: > > > > https://gist.github.com/lojic/892049e617637903f982 > > > > I think my next step will be to create a version that uses places. > > make-shared-bytes may be useful for that. I'll report back with timings. I > > have a 4 core macbook pro w/ 8 hyperthreads - no idea whether the > > hyperthreads are actually useful, but if I can get a 3x speedup with 4 > > cores, I'd be pretty pleased. > > > > I suppose the following is a reasonable architecture: > > > > 1 place for reading the input file and placing a record in a input queue > > N places (one per core) to read from the input queue, process and place in > > an output queue > > 1 place for reading the output queue and writing to either of two output > > files
Thanks. Yes, I have a lot of cleanup to do - I basically hacked this together as fast as I could to experiment. I had wondered about caching the soundex values in the past, so I just coded up a version and concluded it's not helpful, but I think part of that is Racket has some efficiency issues with growing hashes. For the smaller, 200K line, file, there was some improvement, but as I increased the percentage improvement dropped significantly and approached zero. I think the remaining "low hanging fruit" is: 1) Reuse byte string buffers and basically manipulate indexes 2) Parallelize with places -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
