Re: [PATCH 1/6] Modify gcc driver for parallel compilation
On Mon, Aug 24, 2020 at 8:06 PM Giuliano Belinassi wrote: > > Hi, Richi. > > On 08/24, Richard Biener wrote: > > On Fri, Aug 21, 2020 at 12:00 AM Giuliano Belinassi > > wrote: > > > > > > Update the driver for parallel compilation. This process work as > > > follows: > > > > > > When calling gcc, the driver will check if the flag > > > "-fparallel-jobs" was provided by the user. If yes, then we will > > > check what is the desired output, and if it can be parallelized. > > > There are the following cases, which is described: > > > > > > 1. -S or -E was provided: We can't run in parallel, as the output > > >can not be easily merged together into one file. > > > > > > 2. -c was provided: When cc1* forks into multiple processes, it > > >must tell the driver where it stored its generated assembler files. > > >Therefore we pass a hidden "-fsplit-outputs=filename" to the compiler, > > >and we check if "filename" was created by it. If yes, we open it, > > >call assembler for each generated asm file > > >(this file must not be empty), and link them together with > > >partial linking to a single .o file. This process is done for each > > >object file in the argument list. > > > > > > 3. -c was not provided, and the final product will be an binary: Here > > >we proceed exactly as 2., but we avoid doing the partial > > >linking, feeding the generated object files directly into the final > > > link. > > > > > > For that to work, we had to heavily modify how the "execute" function > > > works, extracting common code which is used multiple times, and > > > also detecting when the command is a call to a compiler or an > > > assembler, as can be seen in append_split_outputs. > > > > > > Finally, we added some tests which reflects all cases found when > > > bootstrapping the compiler, so development of further features to the > > > driver get faster for now on. > > > > Few comments inline, Joseph may want to comment on the overall > > structure as driver maintainer (CCed). > > > > I know I asked for the changes on the branch to be squashed but > > the diff below is quite unreadable with the ChangeLog not helping > > the overall refactoring much. Is it possible to do some of the > > factoring/refactoring without any functionality change to make the > > actual diff easier to follow? > > Well, the refactoring is necessary, otherwise I would need to copy and > paste a really huge amount of code. > > What I can do (and sounds reasonable to me) is to break this patch into > two parts; one with just refactoring changes, and the other adding the > parallelism engine. > > > > > Thanks, > > Richard. > > > > > gcc/ChangeLog > > > 2020-08-20 Giuliano Belinassi > > > > > > * common.opt (fsplit-outputs): New flag. > > > (fparallel-jobs): New flag. > > > * gcc.c (extra_arg_storer): New class. > > > (have_S): New variable. > > > (struct command): Move from execute. > > > (is_compiler): New function. > > > (is_assembler): New function. > > > (get_number_of_args): New function. > > > (get_file_by_lines): New function. > > > (identify_asm_file): New function. > > > (struct infile): New attribute temp_additional_asm. > > > (current_infile): New variable. > > > (get_path_to_ld): New function. > > > (has_hidden_E): New function. > > > (sort_asm_files): New function. > > > (append_split_outputs): New function. > > > (print_command): New function. > > > (print_commands): New function. > > > (print_argbuf): New function. > > > (handle_verbose): Extracted from execute. > > > (append_valgrind): Same as above. > > > (async_launch_commands): Same as above. > > > (await_commands_to_finish): Same as above. > > > (split_commands): Same as above. > > > (parse_argbuf): Same as above. > > > (execute): Refator. > > > (fsplit_arg): New function. > > > (alloc_infile): Initialize infiles with 0. > > > (process_command): Remember when -S was passed. > > > (do_spec_on_infiles): Remember current infile being processed. > > > (maybe_run_linker): Replace object files when -o is a executable. > > > (finalize): Deinitialize temp_object_files. > > > > > > gcc/testsuite/ChangeLog: > > > 20-08-2020 Giuliano Belinassi > > > > > > * driver/driver.exp: New test. > > > * driver/a.c: New file. > > > * driver/b.c: New file. > > > * driver/empty.c: New file. > > > * driver/foo.c: New file. > > > --- > > > gcc/common.opt |4 + > > > gcc/gcc.c | 1219 --- > > > gcc/testsuite/driver/a.c|6 + > > > gcc/testsuite/driver/b.c|6 + > > > gcc/testsuite/driver/driver.exp | 80 ++ > > > gcc/testsuite/driver/empty.c|0 > > >
Re: [PATCH 1/6] Modify gcc driver for parallel compilation
Hi, Richi. On 08/24, Richard Biener wrote: > On Fri, Aug 21, 2020 at 12:00 AM Giuliano Belinassi > wrote: > > > > Update the driver for parallel compilation. This process work as > > follows: > > > > When calling gcc, the driver will check if the flag > > "-fparallel-jobs" was provided by the user. If yes, then we will > > check what is the desired output, and if it can be parallelized. > > There are the following cases, which is described: > > > > 1. -S or -E was provided: We can't run in parallel, as the output > >can not be easily merged together into one file. > > > > 2. -c was provided: When cc1* forks into multiple processes, it > >must tell the driver where it stored its generated assembler files. > >Therefore we pass a hidden "-fsplit-outputs=filename" to the compiler, > >and we check if "filename" was created by it. If yes, we open it, > >call assembler for each generated asm file > >(this file must not be empty), and link them together with > >partial linking to a single .o file. This process is done for each > >object file in the argument list. > > > > 3. -c was not provided, and the final product will be an binary: Here > >we proceed exactly as 2., but we avoid doing the partial > >linking, feeding the generated object files directly into the final link. > > > > For that to work, we had to heavily modify how the "execute" function > > works, extracting common code which is used multiple times, and > > also detecting when the command is a call to a compiler or an > > assembler, as can be seen in append_split_outputs. > > > > Finally, we added some tests which reflects all cases found when > > bootstrapping the compiler, so development of further features to the > > driver get faster for now on. > > Few comments inline, Joseph may want to comment on the overall > structure as driver maintainer (CCed). > > I know I asked for the changes on the branch to be squashed but > the diff below is quite unreadable with the ChangeLog not helping > the overall refactoring much. Is it possible to do some of the > factoring/refactoring without any functionality change to make the > actual diff easier to follow? Well, the refactoring is necessary, otherwise I would need to copy and paste a really huge amount of code. What I can do (and sounds reasonable to me) is to break this patch into two parts; one with just refactoring changes, and the other adding the parallelism engine. > > Thanks, > Richard. > > > gcc/ChangeLog > > 2020-08-20 Giuliano Belinassi > > > > * common.opt (fsplit-outputs): New flag. > > (fparallel-jobs): New flag. > > * gcc.c (extra_arg_storer): New class. > > (have_S): New variable. > > (struct command): Move from execute. > > (is_compiler): New function. > > (is_assembler): New function. > > (get_number_of_args): New function. > > (get_file_by_lines): New function. > > (identify_asm_file): New function. > > (struct infile): New attribute temp_additional_asm. > > (current_infile): New variable. > > (get_path_to_ld): New function. > > (has_hidden_E): New function. > > (sort_asm_files): New function. > > (append_split_outputs): New function. > > (print_command): New function. > > (print_commands): New function. > > (print_argbuf): New function. > > (handle_verbose): Extracted from execute. > > (append_valgrind): Same as above. > > (async_launch_commands): Same as above. > > (await_commands_to_finish): Same as above. > > (split_commands): Same as above. > > (parse_argbuf): Same as above. > > (execute): Refator. > > (fsplit_arg): New function. > > (alloc_infile): Initialize infiles with 0. > > (process_command): Remember when -S was passed. > > (do_spec_on_infiles): Remember current infile being processed. > > (maybe_run_linker): Replace object files when -o is a executable. > > (finalize): Deinitialize temp_object_files. > > > > gcc/testsuite/ChangeLog: > > 20-08-2020 Giuliano Belinassi > > > > * driver/driver.exp: New test. > > * driver/a.c: New file. > > * driver/b.c: New file. > > * driver/empty.c: New file. > > * driver/foo.c: New file. > > --- > > gcc/common.opt |4 + > > gcc/gcc.c | 1219 --- > > gcc/testsuite/driver/a.c|6 + > > gcc/testsuite/driver/b.c|6 + > > gcc/testsuite/driver/driver.exp | 80 ++ > > gcc/testsuite/driver/empty.c|0 > > gcc/testsuite/driver/foo.c |7 + > > 7 files changed, 1049 insertions(+), 273 deletions(-) > > create mode 100644 gcc/testsuite/driver/a.c > > create mode 100644 gcc/testsuite/driver/b.c > > create mode 100644 gcc/testsuite/driver/driver.exp > > create mode 100644
Re: [PATCH 1/6] Modify gcc driver for parallel compilation
On Fri, Aug 21, 2020 at 12:00 AM Giuliano Belinassi wrote: > > Update the driver for parallel compilation. This process work as > follows: > > When calling gcc, the driver will check if the flag > "-fparallel-jobs" was provided by the user. If yes, then we will > check what is the desired output, and if it can be parallelized. > There are the following cases, which is described: > > 1. -S or -E was provided: We can't run in parallel, as the output >can not be easily merged together into one file. > > 2. -c was provided: When cc1* forks into multiple processes, it >must tell the driver where it stored its generated assembler files. >Therefore we pass a hidden "-fsplit-outputs=filename" to the compiler, >and we check if "filename" was created by it. If yes, we open it, >call assembler for each generated asm file >(this file must not be empty), and link them together with >partial linking to a single .o file. This process is done for each >object file in the argument list. > > 3. -c was not provided, and the final product will be an binary: Here >we proceed exactly as 2., but we avoid doing the partial >linking, feeding the generated object files directly into the final link. > > For that to work, we had to heavily modify how the "execute" function > works, extracting common code which is used multiple times, and > also detecting when the command is a call to a compiler or an > assembler, as can be seen in append_split_outputs. > > Finally, we added some tests which reflects all cases found when > bootstrapping the compiler, so development of further features to the > driver get faster for now on. Few comments inline, Joseph may want to comment on the overall structure as driver maintainer (CCed). I know I asked for the changes on the branch to be squashed but the diff below is quite unreadable with the ChangeLog not helping the overall refactoring much. Is it possible to do some of the factoring/refactoring without any functionality change to make the actual diff easier to follow? Thanks, Richard. > gcc/ChangeLog > 2020-08-20 Giuliano Belinassi > > * common.opt (fsplit-outputs): New flag. > (fparallel-jobs): New flag. > * gcc.c (extra_arg_storer): New class. > (have_S): New variable. > (struct command): Move from execute. > (is_compiler): New function. > (is_assembler): New function. > (get_number_of_args): New function. > (get_file_by_lines): New function. > (identify_asm_file): New function. > (struct infile): New attribute temp_additional_asm. > (current_infile): New variable. > (get_path_to_ld): New function. > (has_hidden_E): New function. > (sort_asm_files): New function. > (append_split_outputs): New function. > (print_command): New function. > (print_commands): New function. > (print_argbuf): New function. > (handle_verbose): Extracted from execute. > (append_valgrind): Same as above. > (async_launch_commands): Same as above. > (await_commands_to_finish): Same as above. > (split_commands): Same as above. > (parse_argbuf): Same as above. > (execute): Refator. > (fsplit_arg): New function. > (alloc_infile): Initialize infiles with 0. > (process_command): Remember when -S was passed. > (do_spec_on_infiles): Remember current infile being processed. > (maybe_run_linker): Replace object files when -o is a executable. > (finalize): Deinitialize temp_object_files. > > gcc/testsuite/ChangeLog: > 20-08-2020 Giuliano Belinassi > > * driver/driver.exp: New test. > * driver/a.c: New file. > * driver/b.c: New file. > * driver/empty.c: New file. > * driver/foo.c: New file. > --- > gcc/common.opt |4 + > gcc/gcc.c | 1219 --- > gcc/testsuite/driver/a.c|6 + > gcc/testsuite/driver/b.c|6 + > gcc/testsuite/driver/driver.exp | 80 ++ > gcc/testsuite/driver/empty.c|0 > gcc/testsuite/driver/foo.c |7 + > 7 files changed, 1049 insertions(+), 273 deletions(-) > create mode 100644 gcc/testsuite/driver/a.c > create mode 100644 gcc/testsuite/driver/b.c > create mode 100644 gcc/testsuite/driver/driver.exp > create mode 100644 gcc/testsuite/driver/empty.c > create mode 100644 gcc/testsuite/driver/foo.c > > diff --git a/gcc/common.opt b/gcc/common.opt > index 4b08e91859f..4aa3ad8c95b 100644 > --- a/gcc/common.opt > +++ b/gcc/common.opt > @@ -3465,4 +3465,8 @@ fipa-ra > Common Report Var(flag_ipa_ra) Optimization > Use caller save register across calls if possible. > > +fsplit-outputs= > +Common Joined Var(split_outputs) > +-fsplit-outputs= Filename in which current Compilation Unit will > be split to. > + > ; This comment is to ensure we retain the blank line
[PATCH 1/6] Modify gcc driver for parallel compilation
Update the driver for parallel compilation. This process work as
follows:
When calling gcc, the driver will check if the flag
"-fparallel-jobs" was provided by the user. If yes, then we will
check what is the desired output, and if it can be parallelized.
There are the following cases, which is described:
1. -S or -E was provided: We can't run in parallel, as the output
can not be easily merged together into one file.
2. -c was provided: When cc1* forks into multiple processes, it
must tell the driver where it stored its generated assembler files.
Therefore we pass a hidden "-fsplit-outputs=filename" to the compiler,
and we check if "filename" was created by it. If yes, we open it,
call assembler for each generated asm file
(this file must not be empty), and link them together with
partial linking to a single .o file. This process is done for each
object file in the argument list.
3. -c was not provided, and the final product will be an binary: Here
we proceed exactly as 2., but we avoid doing the partial
linking, feeding the generated object files directly into the final link.
For that to work, we had to heavily modify how the "execute" function
works, extracting common code which is used multiple times, and
also detecting when the command is a call to a compiler or an
assembler, as can be seen in append_split_outputs.
Finally, we added some tests which reflects all cases found when
bootstrapping the compiler, so development of further features to the
driver get faster for now on.
gcc/ChangeLog
2020-08-20 Giuliano Belinassi
* common.opt (fsplit-outputs): New flag.
(fparallel-jobs): New flag.
* gcc.c (extra_arg_storer): New class.
(have_S): New variable.
(struct command): Move from execute.
(is_compiler): New function.
(is_assembler): New function.
(get_number_of_args): New function.
(get_file_by_lines): New function.
(identify_asm_file): New function.
(struct infile): New attribute temp_additional_asm.
(current_infile): New variable.
(get_path_to_ld): New function.
(has_hidden_E): New function.
(sort_asm_files): New function.
(append_split_outputs): New function.
(print_command): New function.
(print_commands): New function.
(print_argbuf): New function.
(handle_verbose): Extracted from execute.
(append_valgrind): Same as above.
(async_launch_commands): Same as above.
(await_commands_to_finish): Same as above.
(split_commands): Same as above.
(parse_argbuf): Same as above.
(execute): Refator.
(fsplit_arg): New function.
(alloc_infile): Initialize infiles with 0.
(process_command): Remember when -S was passed.
(do_spec_on_infiles): Remember current infile being processed.
(maybe_run_linker): Replace object files when -o is a executable.
(finalize): Deinitialize temp_object_files.
gcc/testsuite/ChangeLog:
20-08-2020 Giuliano Belinassi
* driver/driver.exp: New test.
* driver/a.c: New file.
* driver/b.c: New file.
* driver/empty.c: New file.
* driver/foo.c: New file.
---
gcc/common.opt |4 +
gcc/gcc.c | 1219 ---
gcc/testsuite/driver/a.c|6 +
gcc/testsuite/driver/b.c|6 +
gcc/testsuite/driver/driver.exp | 80 ++
gcc/testsuite/driver/empty.c|0
gcc/testsuite/driver/foo.c |7 +
7 files changed, 1049 insertions(+), 273 deletions(-)
create mode 100644 gcc/testsuite/driver/a.c
create mode 100644 gcc/testsuite/driver/b.c
create mode 100644 gcc/testsuite/driver/driver.exp
create mode 100644 gcc/testsuite/driver/empty.c
create mode 100644 gcc/testsuite/driver/foo.c
diff --git a/gcc/common.opt b/gcc/common.opt
index 4b08e91859f..4aa3ad8c95b 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -3465,4 +3465,8 @@ fipa-ra
Common Report Var(flag_ipa_ra) Optimization
Use caller save register across calls if possible.
+fsplit-outputs=
+Common Joined Var(split_outputs)
+-fsplit-outputs= Filename in which current Compilation Unit will be
split to.
+
; This comment is to ensure we retain the blank line above.
diff --git a/gcc/gcc.c b/gcc/gcc.c
index 10bc9881aed..c276a11ca7a 100644
--- a/gcc/gcc.c
+++ b/gcc/gcc.c
@@ -343,6 +343,74 @@ static struct obstack obstack;
static struct obstack collect_obstack;
+/* This is used to store new argv arrays created dinamically to avoid memory
+ leaks. */
+
+class extra_arg_storer
+{
+ public:
+
+/* Initialize the vec with a default size. */
+
+extra_arg_storer ()
+ {
+ string_vec.create (8);
+ extra_args.create (64);
+ }
+
+/* Create new array of strings of size N. */
+const char **create_new (size_t n)
+ {
+ const char **ret = XNEWVEC (const char *, n);
+
