Bryan: bash-4.1$ java -version java version "1.6.0_27" Java(TM) SE Runtime Environment (build 1.6.0_27-b07) Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
I tried removing the call to sb.reverse() to see what would happen. I am not seeing much of a difference change in performance. I tried it on words6, which is the original words file contacted 6 times, and I am getting real time values from 3.5s to 4.3s with sb.reverse() commented out, vs not commented out, and it does not seem to make a difference. Do you still think Unicode is the problem. How do you explain the system time of 0.8s, which is roughly 20% of the elapsed time? Well, this is how I explain it. From strace: - first I had to do strace -f to figure it out because the process was forking itself and the real work was being done in a separate thread - inefficient, but for a large app it is OK. However it would be nice if you could disable it and just tell the JVM to do the darn job - is there a switch to do that? - it reads the file reasonably in blocks of 8K - mmap() would be faster, but I do not think this is where you lose by a factor of 20. - however, this is a disaster: 24410 write(1, "0801", 4) = 4 24410 write(1, "\n", 1) = 1 24410 write(1, "tniop-01", 8) = 8 24410 write(1, "\n", 1) = 1 24410 write(1, "ht01", 4) = 4 and that is where most of 0.8s of system time is from, I think. If you buffered your output, you may see much better results. Now, to answer Levi's objections. I did not get offended by your comments, I just felt challenged and excited, and I had fun. I do not think of this exercise as a waste of time. This was very educational for me, I hope it was for others. To make Levi happy - here is the version that does not limit the string size: http://asksasha.com/strrev-fast.c and it runs more than twice as fast the original version as well. It did take me about extra 20 minutes to make it work, a bit longer than I thought (I'll blame it on my Perl experience, it enhanced my natural hubris - one of the three virtues of a programmer). It was very interesting. I thought that if if I just switched from making a copy of the string to be reversed and outputing it with puts() to traversing it backward and outputing each char with putchar() the overhead of putchar() would be compensated by eliminating a call and a string copy with byte swapping. I was wrong - putchar() was so slow that it make the new code run about 1.5 slower than the original. So then I implemented my own buffering, and while doing so I put in a bug that took me about 10 minutes to find. But that did the trick giving a factor of 2 improvement compared to the original. One thing we can learn from this is to get a feel of how fast what you write will run, and how long it takes you do perform different jobs. I think too often we are afraid to consider C when we should - in all honesty because we lack the skill in it, but we explain it by the fear of coredumps, memory leaks, buffer overruns, the lack of primitives, and the general difficulty of getting things done. But here is some math to be derived from this exercise: Perl version runs about 10 times slower than my best C version with the same functionality. For the sake of fairness, let's say the Perl version really took 10 minutes, not 2 to get it to work right. The originally posted version was claimed to have been written in 2 minutes, but I had to fix it up and make sure it worked correctly. I am including in my total of 45 minutes for the C version the validation of correctness as well. So let's just say that in this particular case Perl took 35 minutes less than C. How big will the file have to be on a system that is comparable to my laptop for those 35 minutes to be worth it? strrev-fast processed 30 M file in 89ms, strrev.pl in 905ms. So we get the throughput rates of 337MB/s and 33 MB/s respectively. Now we have the equation: X/337+35*60=X/33 this is a linear equation with ugly numbers, so I just let Wolfram Alpha solve it for me, and I got X = 76823 MB So basically if your file is around 76G or if you have that much in several files, you break even with the C solution in this particular case. If more, you win with C, if less you win with Perl. That is, of course, assuming that you still remember C. Cheating a little bit, now that the C solution has been made public, you just win with the C solution with much smaller data sets because all you need to do now is compile it :-) More seriously, it can be quickly adapted to do other relatively simple file fixing tasks, in particular the ones that require several primitives in Perl and/or Python but are at the core nature simple enough by doing some table lookups and moving around bytes accordingly. Each new primitive you have to call in a scripting language comes at a cost, while in C you are coding your own primitive that just gets slightly more complex, so that type of set up favors C heavily. Levi - I still would like to see some code in Haskell, even if it is dog slow and I have to do some work to get it to run on my box. I programmed in Scheme for a class back in 1994 (CS 330), I do remember writing a program in C to help me track down my mismatched braces, and writing a poem "Count your braces, count them one by one", but that is my extent of exposure to functional programming. -- Sasha Pachev Fast Running Blog. http://fastrunningblog.com Run. Blog. Improve. Repeat. /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
