Re: ftbench update: make integrated
> I have done the changes you want. Thanks! >> 36.5% run difference is bd. AFAICS, you haven't yet worked on >> omitting 'warmup' iterations, right? > > I am planning to increase the iteration count by 10% and ignore the > results for them. You mean you are going to ignore the first 10% of the iterations? This might be a good default. However, I still think that a `--warmup` command-line option makes sense to control this. > Trying to figure out the benchmarking program but actually drowning > in 1500 lines of code. :-) I hope you can eventually find what you need. Werner
Re: ftbench update: make integrated
Hi, I have done the changes you want. > 36.5% run difference is bd. AFAICS, you haven't yet worked on > omitting 'warmup' iterations, right? I am planning to increase the iteration count by 10% and ignore the results for them. Trying to figure out the benchmarking program but actually drowning in 1500 lines of code. Best, Goksu goksu.in On 31 Jul 2023 07:53 +0300, Werner LEMBERG , wrote: > > Here is my new bunch of remarks. In total, we are getting nearer to > meaningful results :-) > > * The build process seems to be ok now, very good! > > * After doing `make baseline` (and `make benchmark`) I see as the last > line > > ``` > Processing 100%...\nBaseline created. > ``` > > This `\n' looks strange. > > * It would be nice if `make benchmark` shows as the last line > something like > > ``` > Benchmark results created in file `/foo/bar/benchmark.html`. > ``` > > * Looking at `benchmark.html`, I'm quite happy now with the layout, > thanks! However, running baseline and benchmark tests for the same > commit on my computer I'm still 'greeted' in the very first test > line with > > ``` > Load 47500 0.167 0.228 -36.5 > ``` > > 36.5% run difference is bd. AFAICS, you haven't yet worked on > omitting 'warmup' iterations, right? > > * In `benchmark.html`, I see > > ``` > *Smaller values mean faster operation > ``` > > please insert a space after '*'. Additionally, you should mark the > columns that are affected by this comment with '*', too, so that the > reader knows what you are referring to. > > > Werner >
Re: ftbench update: make integrated
Here is my new bunch of remarks. In total, we are getting nearer to meaningful results :-) * The build process seems to be ok now, very good! * After doing `make baseline` (and `make benchmark`) I see as the last line ``` Processing 100%...\nBaseline created. ``` This `\n' looks strange. * It would be nice if `make benchmark` shows as the last line something like ``` Benchmark results created in file `/foo/bar/benchmark.html`. ``` * Looking at `benchmark.html`, I'm quite happy now with the layout, thanks! However, running baseline and benchmark tests for the same commit on my computer I'm still 'greeted' in the very first test line with ``` Load 47500 0.167 0.228 -36.5 ``` 36.5% run difference is bd. AFAICS, you haven't yet worked on omitting 'warmup' iterations, right? * In `benchmark.html`, I see ``` *Smaller values mean faster operation ``` please insert a space after '*'. Additionally, you should mark the columns that are affected by this comment with '*', too, so that the reader knows what you are referring to. Werner
Re: ftbench update: make integrated
> about percentages, i runned the bench with -c 200 to have instant > results for development process. here in the benchmark file > attached, it made more acceptable result when increased the -c flag > to 2000. This is much better, thanks! However, there are still tests which have a difference of over 10% for the same commit, inspite of having a very large number of runs. Increasing the number of iterations is actually a brute-force method – what are the timings for the new defaults? The tests must not be too slow, overwise I could run everything in a virtual machine like 'valgrind' and use just a single iteration... Apropos timings: Please add some info to the HTML page that tells how long it takes to test a given font (or perhaps even more detailed information to tell how long it takes to perform a certain test). I suggest that you have a look at other statistical tools that do such sampling, for example google's 'benchmark' project. In particular, have a look at its user manual: https://github.com/google/benchmark/blob/main/docs/user_guide.md What caught my attention especially was the warmup-time option: Maybe it helps if you add an option `--warmup=N` to the benchmark program to make it ignore the first N iterations before starting the timing. Maybe there are other things in the user manual (and/or source code) that you could use to improve the statistical quality of the FreeType tests. Other useful information to reduce the variance can be found here; please do some research of what might be applicable! https://github.com/google/benchmark/blob/main/docs/reducing_variance.md > I changed compiling and the linking process as the demo programs. i > would like to continue to another build system if it seem ok. Will test soon. Werner
Re: ftbench update: make integrated
Thank you Hin-Tak. I have checked the makefile of demos and used libs and the includes as there. (it was overriding the ccraw to cc) about percentages, i runned the bench with -c 200 to have instant results for development process. here in the benchmark file attached, it made more acceptable result when increased the -c flag to 2000. I changed compiling and the linking process as the demo programs. i would like to continue to another build system if it seem ok. Best, Goksu goksu.in On 16 Jul 2023 10:44 +0300, Werner LEMBERG , wrote: > > > * i modified benchmark program not to report 'time per op’ but > > rather 'cumulative time per N iterations' > > * changed the table design > > * sentence 'smaller values are better’ is present > > * embed a small CSS fragment at the top of the page > > * linked to the original baseline and benchmark `.txt` > > * everything is being created in the build directory > > Nice, thanks! Now the next problem: For the same commit IDs, I see > differences in percentage up to 47% in your HTML file! This > essentially means that the delivered numbers are still completely > meaningless – the differences must be at most a few percent or even > smaller, given that the tests are run on exactly the same machine. > > Please investigate how to improve that, probably by modifying the > benchmark test options, or probably even by implementing per-test > options so that the single tests can be fine-tuned. Perhaps you > should do some internet research to find how other, similar benchmark > tests are constructed to get meaningful numbers. > > > Werner Title: Benchmark Results Benchmark Results Warning: Baseline and Benchmark have the same commit ID Info InfoBaselineBenchmark Parameters-c 2000-c 2000 Commit IDe9362ecce9362ecc Commit Date2023-07-14 16:18:00 +03002023-07-14 16:18:00 +0300 BranchGSoC-2023-AhmetGSoC-2023-Ahmet *Smaller values mean faster operation Results for Roboto_subset.ttf TestNBaseline (ms)Benchmark (ms)Difference (%) Load241218.2991146.1145.9 Load_Advances (Normal)241253.1121146.1978.5 Load_Advances (Fast)246.2426.1132.1 Load_Advances (Unscaled)245.7075.780-1.3 Render207120 / 197280785.332779.6170.7 Get_Glyph24355.068347.5082.1 Get_Char_Index1880005.0134.9631.0 Iterate CMap20003.9944.032-1.0 New_Face200085.61486.143-0.6 Embolden24473.296463.5752.1 Stroke55800 / 552001595.6431599.108-0.2 Get_BBox24237.693232.3962.2 Get_CBox24180.251
Re: ftbench update: make integrated
> * i modified benchmark program not to report 'time per op’ but > rather 'cumulative time per N iterations' > * changed the table design > * sentence 'smaller values are better’ is present > * embed a small CSS fragment at the top of the page > * linked to the original baseline and benchmark `.txt` > * everything is being created in the build directory Nice, thanks! Now the next problem: For the same commit IDs, I see differences in percentage up to 47% in your HTML file! This essentially means that the delivered numbers are still completely meaningless – the differences must be at most a few percent or even smaller, given that the tests are run on exactly the same machine. Please investigate how to improve that, probably by modifying the benchmark test options, or probably even by implementing per-test options so that the single tests can be fine-tuned. Perhaps you should do some internet research to find how other, similar benchmark tests are constructed to get meaningful numbers. Werner
Re: ftbench update: make integrated
Cool. You should use "$(CC)" instead of "gcc" explicitly. Read the "implicit variables" section in the GNU manual regarding what built-in variables are there. Ideally you should use "$(LINK_CMD)" there too. That is defined towards the top of the Makefile. Good you have found "echo". You know you can "echo $(CC)" etc to display what values those variables take, too? The first rule likewise should be "$(COMPILE)" to fit in to the rest, in terms of style. Certainly very well done snd congratulations you have found your way around libtool and make! On Friday, 14 July 2023 at 14:26:28 BST, Ahmet Göksu wrote: Thanks a lot, it works now. I have updated the makefile like: # Build ftbench.o $(FTBENCH_OBJ): $(FTBENCH_SRC) @echo "Building ftbench object..." $(CC) $(INCLUDES) -c $< -o $@ # Build ftbench $(FTBENCH_BIN): $(FTBENCH_OBJ) @echo "Linking ftbench..." $(LIBTOOL) --mode=link gcc -L$(LIB_DIR) -lfreetype $< -o $@ @echo "Built." but at the second rule, to produce this line: $(LIBTOOL) --mode=link gcc, i used gcc instead of $(CC) which is an also a libtool. Is it a wrong approach? i remember something about not to using gcc in past mails. Best,Goksugoksu.inOn 12 Jul 2023 22:17 +0300, Hin-Tak Leung , wrote: (Please keep the CC to freetype-devel) libtool is also heavily in the GNU family, although mainly maintained by the people at https://sourceware.org/ , I think. (ancient time people politics...) . http://www.gnu.org/software/libtool/ is also quite well-maintained. You actually don't need to know much about libtool, really; it mostly just launches other commands. i.e. what follows " libtool —mode=compile " is mostly a valid compiler command, you should be able to cut and paste it and run it directly. If it does not work, variable expansions inside your makefile is wrong. So what follows --mode=compile should mostly looks like gcc source1.c -o source1.o or something similar. The "libtool —mode=link " mode is mostly a linker command too. So it should looks like gcc -Llocation1 -Llocation2 -lone -ltwo -Iinclude1 -Iinclude2 1source1.o source2.o source3.o -o binary1 The only difference from a properly valid linking command vs what follows "libtool —mode=link " is that some *.la files happens instead of the equivalent *.so or *.a . For example, I am doing ft2-demo builds at the moment, and the screen echos this commands it is running: libtool --mode=link ... /home/Hin-Tak/git-others/freetype2/objs/libfreetype.la ... in the middle. and it is followed by this information in the next line: libtool: link: ... /home/Hin-Tak/git-others/freetype2/objs/.libs/libfreetype.so ... What follows "libtool: link: " is the actual compiler command it is running. That you should be able to cut and paste and run directly and it should execute without error. Anyway, this is just a quick tutorial... there is no substitute for actually reading the manual :-). They are quite well written, as I keep saying :-). Your "libtool --mode=link " line below is obviously wrong. It is missing the actual compiler name (gcc/cc/clang) itself. On Wednesday, 12 July 2023 at 16:22:04 BST, Ahmet Göksu wrote: thanks a lot, Make works good actually :) the problem i am facing is linking the .o file to binary (even by terminal commands, manually). like: libtool —mode=link ftbench.o -o ftbench some binary, flag or parameter needed i think. Thanks, Goksu goksu.in On 12 Jul 2023 6:12 PM +0300, Hin-Tak Leung , wrote: The GNU Make manual is quite useful - and very well-written too. Go and download it and sit down for an afternoon, away from the keyboard, read sections of it. Good indexes and cross-references too. I think you probably want to do "make -n", for make to just print out what it will do, and just read what all the variables expanding to. It is really just Goal : dependencies How-to-make-goal-from-dependencies There is a section on implicit variables ($@, $^, $<, $CC) and patterns and substitutions, (usage of subst). As I said, it is very well written, and explains a lot of things much better than I or anybody could ever do in a short e-mail :-). On Wednesday, 12 July 2023 at 22:58:47 GMT+8, Ahmet Göksu wrote: actually i am stucked in linking. cc generates object file well but here i am missing something: $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) tried also this $(LIBTOOL) --mode=link $(CC) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) and this $(LIBTOOL) --mode=link $(CC) $^ -o $@ Best, Goksu goksu.in On 12 Jul 2023 5:33 PM +0300, Hin-Tak Leung , wrote: On Wednesday, 12 July 2023 at 21:25:05 GMT+8, Ahmet Göksu wrote: > but i am stucked to binary. $(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) > i am trying to do it same way in the demos, yet didnt figured it out. I haven't been
Re: ftbench update: make integrated
Thanks a lot, it works now. I have updated the makefile like: > > # Build ftbench.o > $(FTBENCH_OBJ): $(FTBENCH_SRC) > @echo "Building ftbench object..." $(CC) $(INCLUDES) -c $< -o $@ > # Build ftbench > $(FTBENCH_BIN): $(FTBENCH_OBJ) > @echo "Linking ftbench..." $(LIBTOOL) --mode=link gcc -L$(LIB_DIR) > -lfreetype $< -o $@ @echo "Built." but at the second rule, to produce this line: $(LIBTOOL) --mode=link gcc, i used gcc instead of $(CC) which is an also a libtool. Is it a wrong approach? i remember something about not to using gcc in past mails. Best, Goksu goksu.in On 12 Jul 2023 22:17 +0300, Hin-Tak Leung , wrote: > (Please keep the CC to freetype-devel) > > libtool is also heavily in the GNU family, although mainly maintained by the > people at https://sourceware.org/ , I think. (ancient time people > politics...) . http://www.gnu.org/software/libtool/ is also quite > well-maintained. > > You actually don't need to know much about libtool, really; it mostly just > launches other commands. i.e. > > what follows " libtool —mode=compile " is mostly a valid compiler > command, you should be able to cut and paste it and run it directly. If it > does not work, variable expansions inside your makefile is wrong. > So what follows --mode=compile should mostly looks like > > gcc source1.c -o source1.o > > or something similar. > > The "libtool —mode=link " mode is mostly a linker command too. So it > should looks like > gcc -Llocation1 -Llocation2 -lone -ltwo -Iinclude1 -Iinclude2 1source1.o > source2.o source3.o -o binary1 > > The only difference from a properly valid linking command vs what follows > "libtool —mode=link " is that some *.la files happens instead of the > equivalent *.so or *.a . For example, I am doing ft2-demo builds at the > moment, and the screen echos this commands it is running: > > libtool --mode=link ... > /home/Hin-Tak/git-others/freetype2/objs/libfreetype.la ... > > in the middle. and it is followed by this information in the next line: > > libtool: link: ... > /home/Hin-Tak/git-others/freetype2/objs/.libs/libfreetype.so ... > > What follows "libtool: link: " is the actual compiler command it is running. > That you should be able to cut and paste and run directly and it should > execute without error. > > > Anyway, this is just a quick tutorial... there is no substitute for actually > reading the manual :-). They are quite well written, as I keep saying :-). > > Your "libtool --mode=link " line below is obviously wrong. It is missing > the actual compiler name (gcc/cc/clang) itself. > > > > On Wednesday, 12 July 2023 at 16:22:04 BST, Ahmet Göksu > wrote: > > > thanks a lot, Make works good actually :) > the problem i am facing is linking the .o file to binary (even by terminal > commands, manually). > > like: > libtool —mode=link ftbench.o -o ftbench > > some binary, flag or parameter needed i think. > > Thanks, > Goksu > goksu.in > On 12 Jul 2023 6:12 PM +0300, Hin-Tak Leung , > wrote: > The GNU Make manual is quite useful - and very well-written too. Go and > download it and sit down for an afternoon, away from the keyboard, read > sections of it. Good indexes and cross-references too. > > I think you probably want to do "make -n", for make to just print out what it > will do, and just read what all the variables expanding to. It is really just > > Goal : dependencies > How-to-make-goal-from-dependencies > > There is a section on implicit variables ($@, $^, $<, $CC) and patterns and > substitutions, (usage of subst). > > As I said, it is very well written, and explains a lot of things much better > than I or anybody could ever do in a short e-mail :-). > > On Wednesday, 12 July 2023 at 22:58:47 GMT+8, Ahmet Göksu > wrote: > > > actually i am stucked in linking. cc generates object file well but here i am > missing something: > $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst > /,$(f),$(FTLIB) $(EFENCE)) > tried also this > $(LIBTOOL) --mode=link $(CC) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) > and this > $(LIBTOOL) --mode=link $(CC) $^ -o $@ > > Best, > Goksu > goksu.in > On 12 Jul 2023 5:33 PM +0300, Hin-Tak Leung , > wrote: > > > On Wednesday, 12 July 2023 at 21:25:05 GMT+8, Ahmet Göksu > wrote: > > > but i am stucked to binary. > $(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) > $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst > /,$(f),$(FTLIB) $(EFENCE)) > > i am trying to do it same way in the demos, yet didnt figured it out. > > I haven't been following your work at all, so I could be wrong. I think you > want to modify the first of the above line to: > > $(OBJ_DIR)/$(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) > ... > > > And elsewhere in the makefile, there should be a pseudo-target of the form: > > all : binary1 binary2 binary3 binary4 > > (in multiple lines continued and separated by "\") > > You want to change that to this sort of
Re: ftbench update: make integrated
(Please keep the CC to freetype-devel) libtool is also heavily in the GNU family, although mainly maintained by the people at https://sourceware.org/ , I think. (ancient time people politics...) . http://www.gnu.org/software/libtool/ is also quite well-maintained. You actually don't need to know much about libtool, really; it mostly just launches other commands. i.e. what follows " libtool —mode=compile " is mostly a valid compiler command, you should be able to cut and paste it and run it directly. If it does not work, variable expansions inside your makefile is wrong. So what follows --mode=compile should mostly looks like gcc source1.c -o source1.o or something similar. The "libtool —mode=link " mode is mostly a linker command too. So it should looks like gcc -Llocation1 -Llocation2 -lone -ltwo -Iinclude1 -Iinclude2 1source1.o source2.o source3.o -o binary1 The only difference from a properly valid linking command vs what follows "libtool —mode=link " is that some *.la files happens instead of the equivalent *.so or *.a . For example, I am doing ft2-demo builds at the moment, and the screen echos this commands it is running: libtool --mode=link ... /home/Hin-Tak/git-others/freetype2/objs/libfreetype.la ... in the middle. and it is followed by this information in the next line: libtool: link: ... /home/Hin-Tak/git-others/freetype2/objs/.libs/libfreetype.so ... What follows "libtool: link: " is the actual compiler command it is running. That you should be able to cut and paste and run directly and it should execute without error. Anyway, this is just a quick tutorial... there is no substitute for actually reading the manual :-). They are quite well written, as I keep saying :-). Your "libtool --mode=link " line below is obviously wrong. It is missing the actual compiler name (gcc/cc/clang) itself. On Wednesday, 12 July 2023 at 16:22:04 BST, Ahmet Göksu wrote: thanks a lot, Make works good actually :) the problem i am facing is linking the .o file to binary (even by terminal commands, manually). like: libtool —mode=link ftbench.o -o ftbench some binary, flag or parameter needed i think. Thanks, Goksu goksu.in On 12 Jul 2023 6:12 PM +0300, Hin-Tak Leung , wrote: The GNU Make manual is quite useful - and very well-written too. Go and download it and sit down for an afternoon, away from the keyboard, read sections of it. Good indexes and cross-references too. I think you probably want to do "make -n", for make to just print out what it will do, and just read what all the variables expanding to. It is really just Goal : dependencies How-to-make-goal-from-dependencies There is a section on implicit variables ($@, $^, $<, $CC) and patterns and substitutions, (usage of subst). As I said, it is very well written, and explains a lot of things much better than I or anybody could ever do in a short e-mail :-). On Wednesday, 12 July 2023 at 22:58:47 GMT+8, Ahmet Göksu wrote: actually i am stucked in linking. cc generates object file well but here i am missing something: $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) tried also this $(LIBTOOL) --mode=link $(CC) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) and this $(LIBTOOL) --mode=link $(CC) $^ -o $@ Best, Goksu goksu.in On 12 Jul 2023 5:33 PM +0300, Hin-Tak Leung , wrote: On Wednesday, 12 July 2023 at 21:25:05 GMT+8, Ahmet Göksu wrote: > but i am stucked to binary. $(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) > i am trying to do it same way in the demos, yet didnt figured it out. I haven't been following your work at all, so I could be wrong. I think you want to modify the first of the above line to: $(OBJ_DIR)/$(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) ... And elsewhere in the makefile, there should be a pseudo-target of the form: all : binary1 binary2 binary3 binary4 (in multiple lines continued and separated by "\") You want to change that to this sort of pattern too: all : $(OBJ_DIR)/binary1 ...
Re: ftbench update: make integrated
The GNU Make manual is quite useful - and very well-written too. Go and download it and sit down for an afternoon, away from the keyboard, read sections of it. Good indexes and cross-references too. I think you probably want to do "make -n", for make to just print out what it will do, and just read what all the variables expanding to. It is really just Goal : dependencies How-to-make-goal-from-dependencies There is a section on implicit variables ($@, $^, $<, $CC) and patterns and substitutions, (usage of subst). As I said, it is very well written, and explains a lot of things much better than I or anybody could ever do in a short e-mail :-). On Wednesday, 12 July 2023 at 22:58:47 GMT+8, Ahmet Göksu wrote: actually i am stucked in linking. cc generates object file well but here i am missing something: $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) tried also this $(LIBTOOL) --mode=link $(CC) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) and this $(LIBTOOL) --mode=link $(CC) $^ -o $@ Best,Goksugoksu.inOn 12 Jul 2023 5:33 PM +0300, Hin-Tak Leung , wrote: On Wednesday, 12 July 2023 at 21:25:05 GMT+8, Ahmet Göksu wrote: > but i am stucked to binary. $(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) > i am trying to do it same way in the demos, yet didnt figured it out. I haven't been following your work at all, so I could be wrong. I think you want to modify the first of the above line to: $(OBJ_DIR)/$(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) ... And elsewhere in the makefile, there should be a pseudo-target of the form: all : binary1 binary2 binary3 binary4 (in multiple lines continued and separated by "\") You want to change that to this sort of pattern too: all : $(OBJ_DIR)/binary1 ...
Re: ftbench update: make integrated
On Wednesday, 12 July 2023 at 21:25:05 GMT+8, Ahmet Göksu wrote: > but i am stucked to binary. $(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst /,$(f),$(FTLIB) $(EFENCE)) > i am trying to do it same way in the demos, yet didnt figured it out. I haven't been following your work at all, so I could be wrong. I think you want to modify the first of the above line to: $(OBJ_DIR)/$(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) ... And elsewhere in the makefile, there should be a pseudo-target of the form: all : binary1 binary2 binary3 binary4 (in multiple lines continued and separated by "\") You want to change that to this sort of pattern too: all : $(OBJ_DIR)/binary1 ...
Re: ftbench update: make integrated
Hi, • i modified benchmark program not to report 'time per op’ but rather 'cumulative time per N iterations' • changed the table design • sentence 'smaller values are better’ is present • embed a small CSS fragment at the top of the page • linked to the original baseline and benchmark `.txt` • everything is being created in the build directory > BASELINE_DIR = $(OBJ_DIR)/baseline/ > BENCHMARK_DIR = $(OBJ_DIR)/benchmark/ • added loading infos • Creating benchmark... • Processing 0-100%… • Benchmark created. and i am stucked here > or the object file creation you have to use `$(CC)`, and for > the final linking you should use `$(LIBTOOL) --mode=link` in normal > unix build mode, and `$(CC)` otherwise. Have a look into the > `Makefile` of the 'freetype-demos' repository for more details. creating obj file works great, > $(OBJ_DIR)/ftbench.$(SO): $(FTBENCH_SRC) > @$(CC) $(INCLUDES) $< -lfreetype -o $@ but i am stucked to binary. > $(FTBENCH_BIN): $(OBJ_DIR)/ftbench.$(SO) > $(LIBTOOL) --mode=link $(CC) $(subst /,$(f),$(LDFLAGS)) $^ -o $@ $(subst > /,$(f),$(FTLIB) $(EFENCE)) i am trying to do it same way in the demos, yet didnt figured it out. Best, Goksu goksu.in On 7 Jul 2023 07:55 +0300, Werner LEMBERG , wrote: > > > > I thought you mean running make outside of the src/tools/ftbench > > > file (as how older version works) by non-sourcedir builds. if you > > > mean running from a seperate build directory, it is working now. > > > > Thanks, will check that soon. > > Well, as it turns out, it doesn't exactly work as a build from a > separate directory should work: The benchmark tests and the HTML page > are still created in the source directory! This must not happen. > Everything must be created in the build directory (except the output > of `./autogen.sh`, but this is not related to your work). > > For fixing this I suggest to use two different (Unix) users: one for > testing, and one for modifying the source code, and the former has no > write access to the latter. You can then immediately identify where > `make baseline` and friends try to write to the source directory. > > Some other remarks. > > * I'm a bit unhappy about > > ``` > Creating baseline... > ``` > > as the only information while the computer churns and grinds, so to > say, for some time. I would rather like to see > > ``` > Creating baseline > foo > bar > ... > Baseline created > ``` > > or something similar. An alternative (or addition) is to show a > percentage of how many fonts have been processed already. > > * Please add links to the original baseline and benchmark `.txt` > files; they contain much more information than displayed on the HTML > page. > > > Werner Title: Benchmark Results Benchmark Results Warning: Baseline and Benchmark have the same commit ID Info InfoBaselineBenchmark Parameters-c 200-c 200 Commit ID9c7800659c780065 Commit Date2023-07-12 15:16:41 +03002023-07-12 15:16:41 +0300 BranchGSoC-2023-AhmetGSoC-2023-Ahmet *Smaller values mean faster operation Results for Roboto_subset.ttf TestNBaseline (ms)Benchmark (ms)Difference (%) Load24000115.541116.820-1.1 Load_Advances (Normal)24000126.435125.9630.4 Load_Advances (Fast)240000.6460.684-5.9 Load_Advances (Unscaled)240000.5690.5680.2 Render2400088.26689.721-1.6 Get_Glyph2400037.03236.5931.2 Get_Char_Index188000.5830.5318.9 Iterate CMap2000.4780.409
Re: ftbench update: make integrated
>> I thought you mean running make outside of the src/tools/ftbench >> file (as how older version works) by non-sourcedir builds. if you >> mean running from a seperate build directory, it is working now. > > Thanks, will check that soon. Well, as it turns out, it doesn't exactly work as a build from a separate directory should work: The benchmark tests and the HTML page are still created in the source directory! This must not happen. Everything must be created in the build directory (except the output of `./autogen.sh`, but this is not related to your work). For fixing this I suggest to use two different (Unix) users: one for testing, and one for modifying the source code, and the former has no write access to the latter. You can then immediately identify where `make baseline` and friends try to write to the source directory. Some other remarks. * I'm a bit unhappy about ``` Creating baseline... ``` as the only information while the computer churns and grinds, so to say, for some time. I would rather like to see ``` Creating baseline foo bar ... Baseline created ``` or something similar. An alternative (or addition) is to show a percentage of how many fonts have been processed already. * Please add links to the original baseline and benchmark `.txt` files; they contain much more information than displayed on the HTML page. Werner
Re: ftbench update: make integrated
> I thought you mean running make outside of the src/tools/ftbench > file (as how older version works) by non-sourcedir builds. if you > mean running from a seperate build directory, it is working now. Thanks, will check that soon. >> * changed .txts to font name and extension >> * added another column that shows the difference in percent >> * added warning message >> * shorten commit id and linking to gitlab >> * parameters and stuff are being printed once Looks good! As can been seen now, the differences vary up to 36% for exactly the same commit, making the output completely meaningless, more or less. Please improve that. A possible solution could be to modify the benchmark program itself: It should not report 'time per op' but rather 'cumulative time per N iterations' so that you have access to more precise values, which can then be used to get better percentages. Another minor improvement might be to change the table design from ``` TestBaselineBenchmark Difference -- Load5.48 us/op 4.90 us/op 10.61% Load_Advances (Normal) -5.65 us/op -5.34 us/op -5.40% Load_Advances (Fast)23.03 us/op 0.03 us/op 21.21% ``` to ``` TestBaseline Benchmark Difference (µs/op) (µs/op)(%) --- Load 5.48 4.90 10.61 Load_Advances (Normal) -5.65 -5.34 -5.40 Load_Advances (Fast)23.03 23.03 21.21 ``` or something similar, thus getting a more horizontally compact layout, and to better align the numbers vertically. BTW, I still miss the sentence 'smaller values are better' (or a variation of it) somewhere at the top. Another peculiarity in your most recent example HTML page is that some lines don't have a green field, but the percentage differs. This is a clear contradiction. Finally (for today :-), I suggest that you embed a small CSS fragment at the top of the page (using a
Re: ftbench update: make integrated
Sorry, my bad to forget to add CC. I thought you mean running make outside of the src/tools/ftbench file (as how older version works) by non-sourcedir builds. if you mean running from a seperate build directory, it is working now. for benchmark.html (also in the attachments): > quote_type > *changed .txts to font name and extension > *added another column that shows the difference in > percent > *added warning message > *shorten commit id and linking to gitlab > *parameters and stuff are being printed once for testing.mk: > quote_type > `make baseline` is working from a separate build directory. > (the error were caused by trying to reach to git details from different > directory, solved it by "git -C $(TOP_DIR)”) > here are the things i stucked: 1) renaming bench.o to bench > quote_type > > quote_type > > i am getting this error when i trim the extension .o: > > quote_type > > > quote_type > > > Building ftbench... > > > libtool: error: cannot determine name of library object from > > > '/Users/gks/Desktop/freetype/objs/bench' > > > make: *** [/Users/gks/Desktop/freetype/objs/bench] Error 1 > quote_type > here is the makefile part related to this. > > > quote_type > > INCLUDES = -I$(TOP_DIR)/include > > > > # Build ftbench > > $(FTBENCH_BIN): $(FTBENCH_SRC) | $(OBJ_DIR) > > @echo "Building ftbench..." > > @$(CC) $(INCLUDES) $< -lfreetype -o $@ 2) multiple compiles of ftbench: > Building ftbench... > ./builds/unix/libtool --mode=compile gcc > -I/Users/gks/Desktop/freetype/include > /Users/gks/Desktop/freetype/src/tools/ftbench/ftbench.c -lfreetype -o > /Users/gks/Desktop/freetype/objs/bench.o > libtool: compile: gcc -I/Users/gks/Desktop/freetype/include > /Users/gks/Desktop/freetype/src/tools/ftbench/ftbench.c -lfreetype > -fno-common -DPIC -o /Users/gks/Desktop/freetype/objs/.libs/bench.o > libtool: compile: gcc -I/Users/gks/Desktop/freetype/include > /Users/gks/Desktop/freetype/src/tools/ftbench/ftbench.c -lfreetype -o > /Users/gks/Desktop/freetype/objs/bench.o >/dev/null 2>&1 i tried both with gcc and it works well. i think i am missing a point related to libtool (parameter?). i need some hint to fix these. Thanks, Goksu goksu.in On 2 Jul 2023 18:55 +0300, Werner LEMBERG , wrote: > > > Is it seem enough to pass mid-term evaluations or is there any other > > improvements needed before the date? > > Looks good in general, and see the other e-mail with a list of tasks. > By the way, it would be nice if you actually did *all* the things I'm > asking for — IIRC, this is the third time I'm talking about > non-sourcedir builds, and it still doesn't work. > > > Werner Title: Benchmark Results Benchmark Results Warning: Baseline and Benchmark have the same commit ID Info InfoBaselineBenchmark Parameters-c 200-c 200 Commit ID1afa56191afa5619 Commit Date2023-07-05 14:00:12 +03002023-07-05 14:00:12 +0300 BranchGSoC-2023-AhmetGSoC-2023-Ahmet Results for Roboto_subset.ttf Test Baseline Benchmark Difference Load5.48 us/op4.90 us/op10.61% Load_Advances (Normal)5.65 us/op5.34 us/op5.40% Load_Advances (Fast)0.03 us/op0.03 us/op21.21% Load_Advances (Unscaled)0.04 us/op0.02 us/op36.84% Render4.07 us/op3.70 us/op9.12% Get_Glyph1.65 us/op1.48 us/op10.61% Get_Char_Index0.03 us/op0.03 us/op-6.90% New_Face54.10 us/op43.62 us/op19.37% Embolden2.07 us/op1.93 us/op6.90% Stroke34.27 us/op30.12 us/op12.13% Get_BBox1.10 us/op0.97 us/op11.67% Get_CBox0.90 us/op0.78 us/op13.56% Results for Arial_subset.ttf Test Baseline Benchmark Difference Load8.33 us/op8.16 us/op1.98% Load_Advances (Normal)9.55 us/op7.87 us/op17.57% Load_Advances (Fast)0.03 us/op0.03 us/op-3.85% Load_Advances (Unscaled)0.03 us/op0.03 us/op0.00% Render4.16 us/op3.77 us/op9.42% Get_Glyph2.08 us/op1.79 us/op13.76% Get_Char_Index0.03 us/op0.03 us/op-3.45% New_Face65.28 us/op52.30 us/op19.89% Embolden2.38 us/op2.19 us/op7.89% Stroke30.57 us/op29.06 us/op4.96% Get_BBox1.18 us/op1.12 us/op5.33% Get_CBox1.01 us/op0.79 us/op22.01% Results for TimesNewRoman_subset.ttf Test Baseline Benchmark Difference Load13.59 us/op9.65 us/op28.99% Load_Advances (Normal)13.02 us/op8.98 us/op31.03% Load_Advances (Fast)0.03 us/op0.03 us/op7.14% Load_Advances (Unscaled)0.03 us/op0.03 us/op3.85% Render4.43 us/op4.42 us/op0.34% Get_Glyph1.86 us/op1.92 us/op-3.22% Get_Char_Index0.03 us/op0.03 us/op-3.23% New_Face61.99 us/op60.97 us/op1.65% Embolden3.66 us/op3.03 us/op17.17% Stroke45.36 us/op39.35 us/op13.25% Get_BBox1.30 us/op1.26 us/op2.62% Get_CBox0.89 us/op0.80 us/op9.70% Results for Tahoma_subset.ttf Test Baseline Benchmark Difference Load5.93 us/op5.06 us/op14.73% Load_Advances (Normal)6.64 us/op5.71 us/op14.06% Load_Advances (Fast)0.03 us/op0.03 us/op0.00% Load_Advances (Unscaled)0.03 us/op0.02 us/op4.00% Render4.18 us/op3.73 us/op10.78% Get_Glyph2.10 us/op1.67 us/op20.59% Get_Char_Index0.03 us/op0.03 us/op6.67% New_Face63.95 us/op52.16 us/op18.44% Embolden2.70 us/op2.36 us/op12.64% Stroke31.08 us/op26.98 us/op13.20% Get_BBox1.36
Re: ftbench update: make integrated
> I am sending this mail as a reminder of last updates that not is > received feedback yet and current GSoC status. Uh, oh, please don't send such e-mails to me privately! You should always address the e-mail list. Note that I'm still in semester-end mode, which means a lot of work for the university. It should be over in two days :-) Additionally it seems that Toshiya-san is very busy, too... > In summary, the benchmark working by Make is in operation so > far. (baseline, benchmark, clean-benchmark) It passes the parameters > FTBENCH_FLAGS=xx and creates a benchmark file that one of its > example is in the attachments. Well, there is still a lot to do :-) * I see, for example, `Roboto_subset.txt` in the generated HTML page. Why `.txt`? Shouldn't the extension be rather `.ttf` (or whatever the actual test file is called)? * I suggest to add another column that shows the difference in percent, for example, '+3.5%' or '-2.7%'. This gives a better overview. * What about adding a warning message at the top (in red) if both the baseline and the benchmark have the same commit ID? This would help users avoid mistakes. * I suggest to shorten the commit IDs similar to what gitlab does; 7 or 8 characters are sufficient. * Please link the commit IDs to gitlab. * Given that all tests use the same bench parameters, commit ID, commit date, and branch names I suggest to move this information to the top, printing it only once. * `make baseline` still doesn't work from a separate build directory. Doing cd ~/git/freetype# the assumed git repository location git clean -fdx ./autogen.sh cd ~ mkdir benchmark cd benchmark ~/git/freetype/configure make -j12 make baseline results in ``` Building ftbench... libtool: compile: gcc -I~/git/freetype/include \ -lfreetype \ ~/git/freetype/src/tools/ftbench/ftbench.c \ -fPIC -DPIC \ -o /home/wl/benchmark/.libs/bench.o libtool: compile: gcc -I~/git/freetype/include \ -lfreetype \ ~/git/freetype/src/tools/ftbench/ftbench.c \ -o /home/wl/benchmark/bench.o \ >/dev/null 2>&1 Creating directory... Creating baseline... fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Baseline created. ``` It's also bad that it says 'Baseline created' for this situation since exactly nothing is created. In other words, you have to abort properly if one or more commands fail. Note that the created binary for the just executed build is called `bench.o`, which is inappropriate. It should be `bench` instead (at least on a GNU/Linux) box. I also see that the bench program is compiled twice, which looks fishy. > I need some advices to future plans. do i need to do absolute same > for meson and cmake? which one first, any other
Re: ftbench update: make integrated
Hello Ahmet, > 1)You can see the benchmarking page in the attachments. Nice! Please also make the HTML page display vital information about the baseline and the benchmark, namely the corresponding git commit IDs (maybe also displaying the commits' date and time). >> Please give more details. What do you consider as 'really much >> time'? You can adjust the number of loops used in `ftbench` with >> command-line options, so it's not clear to me where exactly the >> problem lies. > > I am working with 2 fonts now and took apprx 65 secs. with five > fonts i took around 4 mins. This is indeed too long. However, you are not using any command-line arguments for `ftbench` to reduce the number of tests and/or the number of loops, are you? I will soon check your most recent code changes, thanks. Werner
Re: ftbench update: make integrated
>>- Don't call `gcc` directly! You should rather use `$(CC)` (or >> probably `$(CCexe)`, I'm not sure right now). > > In my understanding, $(CC) can be a cross compiler (e.g. building > Win32 binary on Linux platform), but $(CCexe) is a native compiler > to build "apinames". > > The compiler to build libfreetype is $(CC), thus, using $(CC) would > be safer option (in above example, using $(CCexe) may try to build > Linux binary executable from Win32 binary library). Yep. However, the question is whether it makes sense at all to make the `baseline` target (and friends) work for cross compilation... Werner
Re: ftbench update: make integrated
Hi, - Don't call `gcc` directly! You should rather use `$(CC)` (or probably `$(CCexe)`, I'm not sure right now). In my understanding, $(CC) can be a cross compiler (e.g. building Win32 binary on Linux platform), but $(CCexe) is a native compiler to build "apinames". The compiler to build libfreetype is $(CC), thus, using $(CC) would be safer option (in above example, using $(CCexe) may try to build Linux binary executable from Win32 binary library). Regards, mpsuzuki On 2023/06/15 0:42, Werner LEMBERG wrote: Helo Ahmet, I want to inform about last update about ftbench. Thanks! fonts are in their own directory but 5 fonts takes really much time. Please give more details. What do you consider as 'really much time'? You can adjust the number of loops used in `ftbench` with command-line options, so it's not clear to me where exactly the problem lies. 2) does the html file satisfy the needs? Please post a sample to this list for all the people who are not going to compile your branch. 3) i am planning to move to develop and integrating cmake and meson. is there anything forgotten so far? I don't think so. Here is a list of things that I noticed while inspecting your code. * `testing.mk` - Why `$(BASELINES)` and not `$(BASELINE)`? Are you going to support more than a single baseline? If yes, how shall this work'? But maybe this is just a misunderstanding of what I call a 'baseline': This is the status of the repository (i.e., a certain commit) that is known to work, and that gets compared against some new code (usually in a git branch), checking for differences. - Using `pkg-config` is a bad idea: You are essentially compiling against another installed FreeType version, which is definitely a no-go. The tests must be executed with the uninstalled, just compiled version of the library. - `freetype.mk` already defines `$(PYTHON)`. Don't override this! If necessary you might add a test for python 3. - Please look up the GNU make manual and check the section on order-only prerequisites – this is what you should use for the rules to create directories. - Don't call `gcc` directly! You should rather use `$(CC)` (or probably `$(CCexe)`, I'm not sure right now). - AFAICS, compilation outside the source tree still doesn't work. You have to use `$(OBJ_DIR)` and friends. - Don't call `rm` directly. You should rather use `$(RM)`. 1) benchmark is running by only one font. i have total 5 but takes too long. how many should benchmark work with and should i subset fonts to smaller pieces (like 50 glyphs)? You should define a make variable to hold default command-line arguments for `ftbench`, and which can be overridden while calling `make`. 2) does the html file satisfy the needs? I don't know yet :-) 3) i am planning to move to develop and integrating cmake and meson. is there anything forgotten so far? Perhaps it makes more sense to make the default build work as it should before handling cmake and meson... Werner
Re: ftbench update: make integrated
Helo Ahmet, > I want to inform about last update about ftbench. Thanks! > fonts are in their own directory but 5 fonts takes > really much time. Please give more details. What do you consider as 'really much time'? You can adjust the number of loops used in `ftbench` with command-line options, so it's not clear to me where exactly the problem lies. > 2) does the html file satisfy the needs? Please post a sample to this list for all the people who are not going to compile your branch. > 3) i am planning to move to develop and integrating cmake and >meson. is there anything forgotten so far? I don't think so. Here is a list of things that I noticed while inspecting your code. * `testing.mk` - Why `$(BASELINES)` and not `$(BASELINE)`? Are you going to support more than a single baseline? If yes, how shall this work'? But maybe this is just a misunderstanding of what I call a 'baseline': This is the status of the repository (i.e., a certain commit) that is known to work, and that gets compared against some new code (usually in a git branch), checking for differences. - Using `pkg-config` is a bad idea: You are essentially compiling against another installed FreeType version, which is definitely a no-go. The tests must be executed with the uninstalled, just compiled version of the library. - `freetype.mk` already defines `$(PYTHON)`. Don't override this! If necessary you might add a test for python 3. - Please look up the GNU make manual and check the section on order-only prerequisites – this is what you should use for the rules to create directories. - Don't call `gcc` directly! You should rather use `$(CC)` (or probably `$(CCexe)`, I'm not sure right now). - AFAICS, compilation outside the source tree still doesn't work. You have to use `$(OBJ_DIR)` and friends. - Don't call `rm` directly. You should rather use `$(RM)`. > 1) benchmark is running by only one font. i have total 5 but takes >too long. how many should benchmark work with and should i >subset fonts to smaller pieces (like 50 glyphs)? You should define a make variable to hold default command-line arguments for `ftbench`, and which can be overridden while calling `make`. > 2) does the html file satisfy the needs? I don't know yet :-) > 3) i am planning to move to develop and integrating cmake and meson. >is there anything forgotten so far? Perhaps it makes more sense to make the default build work as it should before handling cmake and meson... Werner
ftbench update: make integrated
Hi, I want to inform about last update about ftbench. Make is integrated with baseline, benchmark and clean-benchmark targets. developed a python script that creates a html file for benchmark too. make baseline: compiles the ftbench.c file and creates a baseline in the src/tools/ftbench/ dir. make benchmark: creates a benchmark in the src/tools/ftbench/ dir and a html file called benchmark.html in top directory. make clean-benchmark: deletes the compiled c file, baseline, benchmark dirs and benchmark.html in top directory. a) src/tools/ftbench/ the ftbench.c file is here. .h file and html creator python script is in src directory. fonts are in their own directory but 5 fonts takes really much time. so i am working with only one font file and the rest are in deactivated dir. b)builds/ created testing.mk and added a line "include $(TOP_DIR)/builds/testing.mk” to freetype.mk. i defined the python as python3 for my machine in testing.mk. edit if that cause any problem. -- 1) benchmark is running by only one font. i have total 5 but takes too long. how many should benchmark work with and should i subset fonts to smaller pieces (like 50 glyphs)? 2) does the html file satisfy the needs? 3) i am planning to move to develop and integrating cmake and meson. is there anything forgotten so far? https://gitlab.freedesktop.org/freetype/freetype/-/tree/GSoC-2023-Ahmet Best, Goksu goksu.in
