I just want to re-emphasize one specific thing: Neither I, nor anyone else, actually owes you any of their time. I don't know you and I don't feel like it is a good investment of my time to convince strangers on the internet. So yes. You asked me to blog about things. I won't and I also won't spend a lot of time and effort trying to dissect your reasoning. I don't feel like that is something I need to apologize for. My choice of wording could have been better, probably, but I still don't feel you have actually answered the question of why the statistical results are not conclusive.
I can only tell you and others why I find your results unconvincing and ultimately meaningless. And you can decide whether you're happy with being unconvincing to me or not (I would've assumed you are fine with that). On Thu, Jun 13, 2019 at 4:47 PM Tong Sun <suntong...@gmail.com> wrote: > Hi Damian & all. > > I think I need to set the record straight, as most of the technical > discussions has been taken place elsewhere, so I feel that I was treated > unfairly because of that. If you only see one site of the story, then it is > very easy to do that, I get it and understand it however. > > So first of all, was I *"like Perl and think it performs good enough or > better for your use"*? No. As I have expressed again and again, I don't > think Go should lag so far behind of Perl. How bad? See for yourself: > > [image: image.png] > > There has some discrepancies and discussions on Apache Bench's statistics, > but my conclusion was that I trust the measured & reported response time > more than the statistical numbers. > > This is the fundamental of the discussion. And as I have expressed again > and again, I want to improve my Go code, because I don't think Go being 8 > times longer than (that *"ridiculously simple")* Perl on average, *"its > results are totally unsurprising"* and should be an accepted as an > *unchangeable > fact*. > > So if you had thought I should be the one who apologies first, expressed > on or off line, please read on... > > On Mon, Jun 10, 2019 at 7:50 PM Axel Wagner wrote: > >> >> On Mon, Jun 10, 2019 at 11:28 PM Tong Sun wrote: >> >>> There is a second test well before your this post, which is a direct >>> translation of Perl code, that is now reading and writing directly to >>> a socket. Hanging on to the first test method and not referring to the >>> second test is not a very constructive way of discussion, let alone >>> using words like "ridiculously ...". >>> >> >> The updated test only changes *both* not to implement a web server. My >> point was, that this is not a realistic test case.e >> >> >>> Over 90% of the code are NOT doing regular expression matching. >>> Focusing *only* on regular expression, not >90% of the rest, is not >>> very convincing but miss the elephant in the room, at least seems to >>> me. >> >> >> The amount of SLOC is not important, however, but where time is spent. >> When adding some basic CPU profiling to the Go code, it spent ~10% of its >> time RE-matching, ~10% in various allocation-related activities, ~75% >> blocking in syscalls and the rest in other various network related runtime >> code. That's pretty much what I meant - in terms of actual CPU time used, >> RE-matching (and some allocation) is pretty much all that's happening. Now, >> I don't know how to profile Perl code and I'll admit that the netpoller >> etc. are part of the language and as such matter for the comparison. But >> the more direct the translation becomes, the less these code-paths will >> differ in performance and the more realistic your scenario gets (in terms >> of concurrency used and required), the more I'd expect Go to be favored by >> it. >> >> Please don't get personal and emotional over technical discussions, >>> and please refrain from using terms like "ridiculously simple" or >>> "reading in tea-leafs" in future discussions. It is inappropriately >>> condescending, thus not the correct attitude of communication. It >>> does not align with the code of conduct of this mlist, and neither of >>> google's, I believe. >>> >>> If you have test results that contradict with mine, please show me >>> yours -- Let's talk the data, and not let the emotion get in the way >>> of technical discussion, and fact the fact, whatever it is. >> >> > There is a reason I'm saying above. Here I quote word for word from Axel > Wagner: > > In any case, in this direct translation, I would've actually assumed perl >> to be faster - because literally all we do is a regexp-match and the perl >> RE-engine is famously optimized and the Go RE-engine is famously meh. That >> perl still ends up with significantly higher average response times is… >> honestly, a confirmation that it's not a very fast language. >> > > >> BTW, I also used a different tool to benchmark (github.com/rakyll/hey). >> AIUI, it uses multiple cores more efficiently. With that tool, the perl >> implementation gets absolutely demolished, with an average response time >> ~40x higher than the Go implementation on my machine. > > > I tried out his implementation eagerly (after thank him publicly of > course), but my result cannot replicate his findings/claims. So I asked him > a follow up question in the blog. He never replies to my inquiry, and now > is this, the only reply I get, from my simple *technical *question, "As > you can see, the test result from my machine is much different from yours. > Would you blog how you tested and your results please?" > > I am not contradicting your data, I'm contradicting your conclusions. >> > > And now he changed his tune. I never accused him of *"reading tea-leafs"* > for coming up the ungrounded conclusion as *"average response time ~40x > higher than Go"*, but he started to blame me for *"reading tea-leafs", * > unprovoked: > > >> *That's why I said "reading tea-leafs"*: You seem to look at the same >> data and come to conclusions that just don't connect for me. For example, >> your original post says that the mean response time for your Perl code >> is 110.875ms resp. 0.222ms and for your Go code it's 83.229ms resp. 0.166ms >> - and yet you claim that your test shows that the mean response time is 8x >> higher for your Go code than your Perl code. >> >> And in your second post, you use `hey`, but then just… throw the results >> out, because you don't understand them (AIUI). >> > > Is it really so? Here is my full reasoning (on the hey part only, taken > from here > <https://dev.to/suntong/simple-web-server-in-perl-and-go-revisit-5d82>): > > ...when I thought more into it, it doesn't make sense to me any more. > > The hey's default settings are: > > Usage: hey [options...] <url> > > > Options: > > -n Number of requests to run. Default is 200. > > -c Number of requests to run concurrently. Total number of requests > cannot > > be smaller than the concurrency level. Default is 50. > > > As we can see in the previous result, the Perl server survives the 500 > concurrent requests gracefully, with 95% percentile reading of only 7ms. > However hey is telling a different story that, with only 200 concurrent > requests, the Perl server's response time is mostly 0.105 seconds. However, > > That is 15 times than ab's reading. > > Hey's inability of getting accurate reading at ms level makes me doubt > it's own accuracy, and I have to suspect that hey's 15-time response time > reading is due to its own slack. > > Moreover, if you take a closer look at hey's Latency distribution > readings, we can see that Perl at 75% is still 0.0181secs, while Go, at 75% > is already 0.0222secs. Let alone at 10%, Perl is 0.0045secs while Go is > 0.0150secs (>3 times more). For this we can see that hey's reading is > contradicting with themselves and Perl clearly performs much better than Go > from hey's Latency distribution readings. > > Furthermore, that 13 long 1.043sec response time from the Perl server is > only observed in hey, all the other performance testing software, including > the not-documented-here httperf, none of them have exhibited such behavior. > So it is fair to conclude that the long 1.043sec response time is caused by > hey itself, not by the Perl server. > > So, up to now, for all above three reasons, I have to conclude that hey is > not suitable for *this test* due to its own limitations, thus cannot > stacks up against the most popular Apache Bench. > > Did I just *"throw the results out"*? I don't personally think so. > And my whole reasoning and discussion were rejected simply with a claim: > *"**throw the results out **because you don't understand them (AIUI)."* > > I just don't know how I'm supposed to handle that discrepancy. All of this >> just looks like really bad science to me. ISTM that defining the criteria >> beforehand and designing the test to measure exactly those criteria is the >> most reasonable way to reconcile that. If we use "mean response time over >> X", it seems that so far your Go code is coming out ahead. And note that >> *I* didn't make any claims about the relative strengths of programming >> languages here. IMO the onus is clearly on you to show your work when >> making these claims. I don't actually care. >> >> > Anyway, all of that being said: While I don't think the test you >>> devised allows the broad generalizations you are making, ultimately I don't >>> have any stakes in this (which is why I haven't responded further on the >>> blog post). If you like Perl and think it performs good enough or better >>> for your use case - then go ahead and use Perl. No one here will begrudge >>> you for it. >>> >>> As I've comment in the blog many times, I'm not trying to prove Perl >>> performs better than Go. On the contrary, I was trying to improve Perl >>> performs with Go, that's why the whole thing get started, as I have >>> replied to your comment in the blog:"that's why I was rewriting the >>> Perl code to Go". >>> >>> It is still the case, and I want to try out everyone's suggestion to >>> see how things can improve. >>> >>> Moreover, if you have taken a look at my second test, you will >>> understand my other goal is to make it clear "who and what to trust". >>> Because if you have taken a look at the httperf test result I posted >>> to this mlist before you reply, you may have realized that, of all the >>> performance testing tools I've used so far, all are suggesting Perl >>> performs better than Go, except for httperf. So maybe the performance >>> testing tools are biased toward Go, and I made it clear in my second >>> blog that I want to get to the bottom of it. >>> >>> > All of that being said >>> > >>> > IMO this is just not a meaningful test. Or its results are totally >>> unsurprising. >>> >>> Again, I was trying to improve Perl performs with Go. That >>> "ridiculously simple" Perl code is the foundation of the Debian dbab >>> package, and I was trying to improve it. >>> >>> It might be meaningless to you but it is perfectly meaningful to me. >>> Please don't be so judgmental. >>> >> > As I have promised, I will try out everyone's suggestion to see how > things can improve, and I did, and I finally had made my Go faster than my > Perl. > > I still think somebody owns me a public apology, but I don't have any hope > for that, and I don't care either. > > >>> > On Sun, Jun 9, 2019 at 4:54 AM Justin Israel <justinisr...@gmail.com> >>> wrote: >>> >> >>> >> I'm wondering about a couple factors in this comparison that seem to >>> make a difference in my local test: >>> >> >>> >> I think perl sockets are write buffered. So would the equivalent be >>> to wrap the net.Conn in bufio.NewWriter(c) and flush before the Close? >>> >> Since this is a straigh-line test where both servers are not using >>> concurrent handling of connections (uncommon for a Go server even on 1 >>> core), would it not make sense to run the Go server with GOMAXPROCS=1? >>> >> >>> >> - Justin >>> >> >>> >> On Saturday, June 8, 2019 at 1:36:49 AM UTC+12, Tong Sun wrote: >>> >>> >>> >>> I had always believed that the web projects build with Go should be >>> much faster than Perl, since Go is a compiled language. >>> >>> >>> >>> However that belief was crushed brutally last night, when I did a >>> comparison -- the Go implementation is 8 times worse than the Perl! -- the >>> mean response time jumped from 6ms to 48ms. >>> >>> >>> >>> I know this is the simplest possible web server, but still, when it >>> comes to simple web servers like this, I have to say that Perl performs >>> much better than Go. >>> >>> >>> >>> I don't think there is much I can twist on the Go side, since it >>> can't be more simpler than that. However, I also believe it won't hurt to >>> ask and confirm. So, >>> >>> >>> >>> Have I missed anything? Is it possible for me to make my Go >>> implementation anywhere near the Perl's performance? >>> >>> >>> >>> Thanks >>> >>> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAMmz1OchP86yyHo_QF-fzOyn%3DaSJvKuTYdvkfRYwO9JPOs3ONg%40mail.gmail.com > <https://groups.google.com/d/msgid/golang-nuts/CAMmz1OchP86yyHo_QF-fzOyn%3DaSJvKuTYdvkfRYwO9JPOs3ONg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGomabZ6W%3DByUbT1d%3D5j5rHBfy63PRFc6qy5myHQwE%2BbA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
