Hi Freddy,
I just ran some further benchmarks between your first and second script,
compared to mine, and again similar results were found. Your second
script was significantly faster than the first, but still didn't match
the grep-piped-into-awk config.
This shouldn't be the case though. I did further testing on my PowerMac
G4 500Mhz workstation running 6.2, which I chose because I thought a
single core ppc G4 500Mhz vs a mips64 dual core 500Mhz would be a pretty
epic showdown. I ran each script twice and wrote the output to /dev/null
to ensure disk I/O wasn't a factor. The StevenBlack hosts file has on
average ~47,000 lines including comments.
The results were somewhat surprising:
The G4 cranked out the scripts with these times:
*Your 1st script: an average of 1.415 seconds**
*
*Your 2nd script: an average of 0.54 seconds*
*My script: an average of 1.71 seconds*
This clearly shows things the way they are supposed to be, with my
script being grossly inefficient and yours being clearly superior. See
below for the times on the Edgerouter Lite:
(Note: All tested times are slower than previous results from last email
due to the machine being under a modest network load during testing.
Load remained consistent due to it being a long running slow 5 megabit
bulk network transfer it was routing. This was unavoidable due to it
being a production machine.)
*Your first script came in at an average of 20.8 seconds*
*Your second script came in at an average of 13.75 seconds *
*And my script came in at an average of 10.25 seconds. *
These results are shockingly poor compared to a G4 of the same clock
speed. The leads me to believe there may be some Octeon specific
inefficiencies at play here, namely floating point. None of the
Edgerouter units have an FPU I believe ( I know for sure the Lite
doesn't) and I am wondering if awk makes heavy use of floating point,
and thus it having to abuse the emulated fpu? During the all awk
scripts, the ERLite becomes cpu bound on 1 core.
It would be awesome if an awk guru here could confirm whether awk makes
heavy use of the fpu.
If this is indeed the case, then the PowerPC would have an extreme
advantage with its beefy AltiVec unit.
So I suppose for those folks running my addblocking scripts, it would be
wise to use Freddy Dissaux's all awk hostfile conversion method if
you're running a more conventional architecture. It would be great if
someone here could post some test results on an arm64 board!
I am now very curious to see how Perl compares against these results. I
hope I can find the time to play around with making a nice optimized
script.
On 12/31/17 03:41, Freddy DISSAUX wrote:
Hello Jordan,
I have tried using all awk for the script before, but I find piping the
grep output into awk to be 2-3x faster on the Edgerouter Lite. I just
ran some timed tests for your script against mine on the ErLite, and I
got similar results, with my script completing in ~6 seconds against the
StevenBlack hosts file, and yours at ~14 seconds. This may not be the
case on more conventional architectures. I am considering rewriting the
script in Perl to see if that runs any faster.
Could you try
awk 'BEGIN { OFS = "" } NF == 2 && $1 == "0.0.0.0" { print "local-zone: \"", $2, "\" redirect"; print
"local-data: \"", $2, " A 0.0.0.0\"" }' hosts > ads.conf
If i understand my tests, 2 print without concat are faster than
1 print with concat (and faster than 1 printf)
cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data:
\""$2" A 0.0.0.0\""}' > ads.conf
UUOC:
grep '^0\.0\.0\.0' host | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A
0.0.0.0\""}' > ads.conf
Regards,
