[Cocci] [PATCH V2 2/2] docs: manual: Add option description in spatch_options
Add documentation for the "use-patch-diff" option introduced in the first patch of this patchset. This option allows for applying a semantic patch to only those files in a directory where code additions have been made. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Change "use-patchdiff" to "use-patch-diff" (Julia) --- docs/manual/spatch_options.tex | 5 + 1 file changed, 5 insertions(+) diff --git a/docs/manual/spatch_options.tex b/docs/manual/spatch_options.tex index a372aaf8a..393d254e3 100644 --- a/docs/manual/spatch_options.tex +++ b/docs/manual/spatch_options.tex @@ -220,6 +220,11 @@ intereted relative to the target directory. If the filename is an absolute path name, beginning with /, it is used as is. } +\normal{-{}-use-patch-diff}{ This option allows for applying a semantic patch + to only the files that have been modified in a directory. The directory needs + to be specified by the user. Note that an absolute path needs to be used if + spatch is called from outside the target project directory. } + \normal{-{}-use-coccigrep}{ Use a version of grep implemented in Coccinelle to check that selected files are relevant to the semantic patch. This option is only relevant to the case of working on a complete directory, -- 2.32.0 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH V2 1/2] parsing_cocci: Add feature to check only modified files
This patch adds an option "use-patch-diff" to allow for applying a semantic patch to only those files in a directory where code additions have been made. Usage: spatch -D --sp-file --use-patch-diff Example: spatch -D report --sp-file for_each_child.cocci --use-patch-diff drivers/gpu Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Change "use-patchdiff" to "use-patch-diff" (Julia) --- Makefile| 2 +- enter.ml| 17 +++-- globals/flag.ml | 2 +- globals/flag.mli| 2 +- ocaml/coccilib.mli | 1 + parsing_cocci/get_constants2.ml | 1 + parsing_cocci/patch_diff.ml | 118 parsing_cocci/patch_diff.mli| 9 +++ 8 files changed, 143 insertions(+), 9 deletions(-) create mode 100755 parsing_cocci/patch_diff.ml create mode 100644 parsing_cocci/patch_diff.mli diff --git a/Makefile b/Makefile index f8d3424c0..455e92395 100644 --- a/Makefile +++ b/Makefile @@ -44,7 +44,7 @@ SOURCES_parsing_cocci := \ parse_printf.ml parse_aux.ml cleanup_rules.ml disjdistr.ml \ parser_cocci_menhir.mly lexer_cocci.mll \ lexer_cli.mll lexer_script.mll \ - cocci_grep.ml dpll.ml get_constants2.ml id_utils.ml git_grep.ml \ + cocci_grep.ml dpll.ml get_constants2.ml id_utils.ml git_grep.ml patch_diff.ml \ adjacency.ml commas_on_lists.ml re_constraints.ml parse_cocci.ml \ command_line.ml cocci_args.ml SOURCES_parsing_c := \ diff --git a/enter.ml b/enter.ml index 77e1540fd..1219c027d 100644 --- a/enter.ml +++ b/enter.ml @@ -18,9 +18,7 @@ module Inc = Includes * globals/config.ml, mainly a standard.h and standard.iso file *) let cocci_file = ref "" - let opt_c_files = ref [] - let output_file = ref "" (* resulting code *) let tmp_dir = ref "" (* temporary files for parallelism *) let aux_file_suffix = @@ -283,7 +281,6 @@ let print_version () = let short_options = [ "--sp-file", Arg.Set_string cocci_file, " the semantic patch file"; - "--opt-c", Arg.String (fun filename -> if Sys.file_exists filename @@ -370,6 +367,9 @@ let short_options = [ "--use-coccigrep", Arg.Unit (function _ -> Flag.scanner := Flag.CocciGrep), "find relevant files using cocci grep"; + "--use-patch-diff", + Arg.Unit (function _ -> Flag.scanner := Flag.PatchDiff), + "process files in the diff for a directory"; "--patch", Arg.String (function s -> Flag.patch := Some (Cocci.normalize_path s)), (" path name with respect to which a patch should be created\n"^ @@ -940,13 +940,18 @@ let idutils_filter (_,_,_,query) dir = Some (files +> List.filter - (fun file -> List.mem (Common.filesuffix file) suffixes)) + (fun file -> List.mem (Common.filesuffix file) suffixes)) + +let patchdiff_filter _ dir = + let struc = Patch_diff.getpatchdiff dir in + Some (List.map (function x -> x.Patch_diff.file_name) struc) let scanner_to_interpreter = function Flag.Glimpse -> glimpse_filter | Flag.IdUtils -> idutils_filter | Flag.CocciGrep -> coccigrep_filter | Flag.GitGrep -> gitgrep_filter + | Flag.PatchDiff -> patchdiff_filter | _ -> failwith "impossible" (*) @@ -1079,7 +1084,7 @@ let rec main_action xs = " or multiple files") | _, false, _, _, _ -> [List.map (fun x -> (x,None)) (x::xs)] | _, true, "", - (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep), + (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep|Flag.PatchDiff), [] -> let interpreter = scanner_to_interpreter !Flag.scanner in let files = @@ -1088,7 +1093,7 @@ let rec main_action xs = | Some files -> files in files +> List.map (fun x -> [(x,None)]) | _, true, s, - (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep), _ + (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep|Flag.PatchDiff), _ when s <> "" -> failwith "--use-xxx filters do not work with --kbuild" (* normal *) diff --git a/globals/flag.ml b/globals/flag.ml index e1d01cb4c..dffe6cd80 100644 --- a/globals/flag.ml +++ b/globals/flag.ml @@ -16,7 +16,7 @@ let track_iso_usage = ref false let worth_trying_opt = ref true -type scanner = IdUtils | Glimpse | CocciGrep | GitGrep | NoScanner +type scanner = IdUtils | Glimpse | CocciGrep | GitGrep | PatchDiff | NoScanner let scanner = ref NoScanner let pyoutp
[Cocci] [PATCH V2 0/2] Add "use-patch-diff" option
This patchset adds a feature to enable Coccinelle to only check all those files in a directory which were modified. It parses all the files obtained from the output of "git diff" and checks them against the specified cocci script. An example for passing the "use-patchdiff" option is: spatch -D report --sp-file for_each_child.cocci --use-patchdiff drivers/gpu One thing to note while using the command is that while it can be invoked outside the target project directory, an absolute path to the target folder should be specified in such cases. Sumera Priyadarsini (2): parsing_cocci: Add feature to check only modified files docs: manual: Add option description in spatch_options Makefile| 2 +- docs/manual/spatch_options.tex | 5 ++ enter.ml| 17 +++-- globals/flag.ml | 2 +- globals/flag.mli| 2 +- ocaml/coccilib.mli | 1 + parsing_cocci/get_constants2.ml | 1 + parsing_cocci/patch_diff.ml | 118 parsing_cocci/patch_diff.mli| 9 +++ 9 files changed, 148 insertions(+), 9 deletions(-) create mode 100755 parsing_cocci/patch_diff.ml create mode 100644 parsing_cocci/patch_diff.mli -- 2.32.0 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 0/2] Add "use-patchdiff" option
On Wed, May 26, 2021 at 10:13 PM Markus Elfring wrote: > > > This patchset adds a feature to enable Coccinelle > > to only check all those files in a directory which were > > modified. It parses all the files obtained from the > > output of "git diff" and checks them against the specified > > cocci script. > > > > An example for passing the "use-patchdiff" option is: > > How do you think about to use the parameter name “use-files-from-diff”? I would prefer something shorter, like "use-filesdiff" but I am okay with either name as long as the maintainers are okay with it. :) Julia, what do you think? I will send a v2 with any of the above name options (and/or any other changes that are suggested.) Thanks, Sumera > > Regards, > Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 2/2] docs: manual: Add option description in spatch_options
Add documentation for the "use-patchdiff" option introduced in the first patch of this patchset. This option allows for applying a semantic patch to only those files in a directory where code additions have been made. Signed-off-by: Sumera Priyadarsini --- docs/manual/spatch_options.tex | 5 + 1 file changed, 5 insertions(+) diff --git a/docs/manual/spatch_options.tex b/docs/manual/spatch_options.tex index 223365bb5..afc55aa22 100644 --- a/docs/manual/spatch_options.tex +++ b/docs/manual/spatch_options.tex @@ -217,6 +217,11 @@ intereted relative to the target directory. If the filename is an absolute path name, beginning with /, it is used as is. } +\normal{-{}-use-patchdiff}{ This option allows for applying a semantic patch + to only the files that have been modified in a directory. The directory needs + to be specified by the user. Note that an absolute path needs to be used if + spatch is called from outside the target project directory. } + \normal{-{}-use-coccigrep}{ Use a version of grep implemented in Coccinelle to check that selected files are relevant to the semantic patch. This option is only relevant to the case of working on a complete directory, -- 2.31.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 1/2] parsing_cocci: Add feature to check only modified files
This patch adds an option "use-patchdiff" to allow for applying a semantic patch to only those files in a directory where code additions have been made. Usage: spatch -D --sp-file --use-patchdiff Example: spatch -D report --sp-file for_each_child.cocci --use-patchdiff drivers/gpu Signed-off-by: Sumera Priyadarsini --- Makefile| 2 +- enter.ml| 17 +++-- globals/flag.ml | 2 +- globals/flag.mli| 2 +- ocaml/coccilib.mli | 1 + parsing_cocci/get_constants2.ml | 1 + parsing_cocci/patch_diff.ml | 118 parsing_cocci/patch_diff.mli| 9 +++ 8 files changed, 143 insertions(+), 9 deletions(-) create mode 100755 parsing_cocci/patch_diff.ml create mode 100644 parsing_cocci/patch_diff.mli diff --git a/Makefile b/Makefile index f8d3424c0..455e92395 100644 --- a/Makefile +++ b/Makefile @@ -44,7 +44,7 @@ SOURCES_parsing_cocci := \ parse_printf.ml parse_aux.ml cleanup_rules.ml disjdistr.ml \ parser_cocci_menhir.mly lexer_cocci.mll \ lexer_cli.mll lexer_script.mll \ - cocci_grep.ml dpll.ml get_constants2.ml id_utils.ml git_grep.ml \ + cocci_grep.ml dpll.ml get_constants2.ml id_utils.ml git_grep.ml patch_diff.ml \ adjacency.ml commas_on_lists.ml re_constraints.ml parse_cocci.ml \ command_line.ml cocci_args.ml SOURCES_parsing_c := \ diff --git a/enter.ml b/enter.ml index 9643df782..587071aa8 100644 --- a/enter.ml +++ b/enter.ml @@ -18,9 +18,7 @@ module Inc = Includes * globals/config.ml, mainly a standard.h and standard.iso file *) let cocci_file = ref "" - let opt_c_files = ref [] - let output_file = ref "" (* resulting code *) let tmp_dir = ref "" (* temporary files for parallelism *) let aux_file_suffix = @@ -283,7 +281,6 @@ let print_version () = let short_options = [ "--sp-file", Arg.Set_string cocci_file, " the semantic patch file"; - "--opt-c", Arg.String (fun filename -> if Sys.file_exists filename @@ -370,6 +367,9 @@ let short_options = [ "--use-coccigrep", Arg.Unit (function _ -> Flag.scanner := Flag.CocciGrep), "find relevant files using cocci grep"; + "--use-patchdiff", + Arg.Unit (function _ -> Flag.scanner := Flag.PatchDiff), + "process files in the diff for a directory"; "--patch", Arg.String (function s -> Flag.patch := Some (Cocci.normalize_path s)), (" path name with respect to which a patch should be created\n"^ @@ -940,13 +940,18 @@ let idutils_filter (_,_,_,query) dir = Some (files +> List.filter - (fun file -> List.mem (Common.filesuffix file) suffixes)) + (fun file -> List.mem (Common.filesuffix file) suffixes)) + +let patchdiff_filter _ dir = + let struc = Patch_diff.getpatchdiff dir in + Some (List.map (function x -> x.Patch_diff.file_name) struc) let scanner_to_interpreter = function Flag.Glimpse -> glimpse_filter | Flag.IdUtils -> idutils_filter | Flag.CocciGrep -> coccigrep_filter | Flag.GitGrep -> gitgrep_filter + | Flag.PatchDiff -> patchdiff_filter | _ -> failwith "impossible" (*) @@ -1079,7 +1084,7 @@ let rec main_action xs = " or multiple files") | _, false, _, _, _ -> [List.map (fun x -> (x,None)) (x::xs)] | _, true, "", - (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep), + (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep|Flag.PatchDiff), [] -> let interpreter = scanner_to_interpreter !Flag.scanner in let files = @@ -1088,7 +1093,7 @@ let rec main_action xs = | Some files -> files in files +> List.map (fun x -> [(x,None)]) | _, true, s, - (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep), _ + (Flag.Glimpse|Flag.IdUtils|Flag.CocciGrep|Flag.GitGrep|Flag.PatchDiff), _ when s <> "" -> failwith "--use-xxx filters do not work with --kbuild" (* normal *) diff --git a/globals/flag.ml b/globals/flag.ml index e1d01cb4c..dffe6cd80 100644 --- a/globals/flag.ml +++ b/globals/flag.ml @@ -16,7 +16,7 @@ let track_iso_usage = ref false let worth_trying_opt = ref true -type scanner = IdUtils | Glimpse | CocciGrep | GitGrep | NoScanner +type scanner = IdUtils | Glimpse | CocciGrep | GitGrep | PatchDiff | NoScanner let scanner = ref NoScanner let pyoutput = ref "coccilib.output.Console" diff --git a/globals/flag.mli b/globals/fl
[Cocci] [PATCH 0/2] Add "use-patchdiff" option
This patchset adds a feature to enable Coccinelle to only check all those files in a directory which were modified. It parses all the files obtained from the output of "git diff" and checks them against the specified cocci script. An example for passing the "use-patchdiff" option is: spatch -D report --sp-file for_each_child.cocci --use-patchdiff drivers/gpu One thing to note while using the command is that while it can be invoked outside the target project directory, an absolute path to the target folder should be specified in such cases as. Sumera Priyadarsini (2): parsing_cocci: Add feature to check only modified files docs: manual: Add option description in spatch_options Makefile| 2 +- docs/manual/spatch_options.tex | 5 ++ enter.ml| 17 +++-- globals/flag.ml | 2 +- globals/flag.mli| 2 +- ocaml/coccilib.mli | 1 + parsing_cocci/get_constants2.ml | 1 + parsing_cocci/patch_diff.ml | 118 parsing_cocci/patch_diff.mli| 9 +++ 9 files changed, 148 insertions(+), 9 deletions(-) create mode 100755 parsing_cocci/patch_diff.ml create mode 100644 parsing_cocci/patch_diff.mli -- 2.31.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2] Documentation: Coccinelle: Improve command example for debugging patches
Modify Coccinelle documentation to clarify usage of make command to run coccicheck on a folder. Changes in v2: - Give example of folder instead of file - Add note Signed-off-by: Sumera Priyadarsini --- Documentation/dev-tools/coccinelle.rst | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 74c5e6aeeff5..9c454de5a7f7 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -224,14 +224,21 @@ you may want to use:: rm -f err.log export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci -make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c +make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd err.log will now have the profiling information, while stdout will provide some progress information as Coccinelle moves forward with work. +NOTE: + DEBUG_FILE support is only supported when using coccinelle >= 1.0.2. +Currently, DEBUG_FILE support is only available to check folders, and +not single files. This is because checking a single file requires spatch +to be called twice leading to DEBUG_FILE being set both times to the same value, +giving rise to an error. + .cocciconfig support -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v3] scripts: coccicheck: Correct usage of make coccicheck
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the error: ./scripts/coccicheck: line 65: -1: shift count out of range This happens because every time the C variable is specified, the shell arguments need to be "shifted" in order to take only the last argument, which is the C file to test. These shell arguments mostly comprise flags that have been set in the Makefile. However, when coccicheck is specified in the make command as a rule, the number of shell arguments is zero, thus passing the invalid value -1 to the shift command, resulting in an error. Modify coccicheck to print correct usage of make coccicheck so as to avoid the error. Signed-off-by: Sumera Priyadarsini --- Changes in v2: - Move test to only display error message Changes in v3: - Update example with latest file --- scripts/coccicheck | 12 1 file changed, 12 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index 209bb0427b43..d1aaa1dc0a69 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -61,6 +61,18 @@ COCCIINCLUDE=${COCCIINCLUDE// -include/ --include} if [ "$C" = "1" -o "$C" = "2" ]; then ONLINE=1 +if [[ $# -le 0 ]]; then + echo '' + echo 'Specifying both the variable "C" and rule "coccicheck" in the make +command results in a shift count error.' + echo '' + echo 'Try specifying "scripts/coccicheck" as a value for the CHECK variable instead.' + echo '' + echo 'Example: make C=2 CHECK=scripts/coccicheck drivers/net/ethernet/ethoc.o' + echo '' + exit 1 +fi + # Take only the last argument, which is the C file to test shift $(( $# - 1 )) OPTIONS="$COCCIINCLUDE $1" -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH v2] scripts: coccicheck: Correct usage of make coccicheck
On Tue, Nov 24, 2020 at 2:36 AM Julia Lawall wrote: > > > On Wed, 18 Nov 2020, Sumera Priyadarsini wrote: > > > The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the > > error: > > ./scripts/coccicheck: line 65: -1: shift count out of range > > > > This happens because every time the C variable is specified, > > the shell arguments need to be "shifted" in order to take only > > the last argument, which is the C file to test. These shell arguments > > mostly comprise flags that have been set in the Makefile. However, > > when coccicheck is specified in the make command as a rule, the > > number of shell arguments is zero, thus passing the invalid value -1 > > to the shift command, resulting in an error. > > > > Modify coccicheck to print correct usage of make coccicheck so as to > > avoid the error. > > > > Signed-off-by: Sumera Priyadarsini > > --- > > Changes in v2: > > - Move test to only display error message > > --- > > scripts/coccicheck | 12 > > 1 file changed, 12 insertions(+) > > > > diff --git a/scripts/coccicheck b/scripts/coccicheck > > index 209bb0427b43..f3b8bf505c5f 100755 > > --- a/scripts/coccicheck > > +++ b/scripts/coccicheck > > @@ -61,6 +61,18 @@ COCCIINCLUDE=${COCCIINCLUDE// -include/ --include} > > if [ "$C" = "1" -o "$C" = "2" ]; then > > ONLINE=1 > > > > +if [[ $# -le 0 ]]; then > > + echo '' > > + echo 'Specifying both the variable "C" and rule "coccicheck" > in the make > > +command results in a shift count error.' > > + echo '' > > + echo 'Try specifying "scripts/coccicheck" as a value for the > CHECK variable instead.' > > + echo '' > > + echo 'Example: make C=2 CHECK=scripts/coccicheck > drivers/staging/wfx/hi_t.o' > > I think that this file doesn't exist any more. Even though you can't > guarantee that any particular file will be around forever, a file in > staging is probably not a good choice, since they are usually on their way > in or on the way out of the kernel. > Fair, I will send a v3 with a better example. regards, sumera > julia > > > + echo '' > > + exit 1 > > +fi > > + > > # Take only the last argument, which is the C file to test > > shift $(( $# - 1 )) > > OPTIONS="$COCCIINCLUDE $1" > > -- > > 2.25.1 > > > > ___ > > Cocci mailing list > > Cocci@systeme.lip6.fr > > https://systeme.lip6.fr/mailman/listinfo/cocci > > > ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] Documentation: Coccinelle: Improve command example for debugging patches
Modify Coccinelle documentation to clarify usage of make command to run coccicheck on a single file. Signed-off-by: Sumera Priyadarsini --- Documentation/dev-tools/coccinelle.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 74c5e6aeeff5..9e60cf175fd6 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -224,7 +224,7 @@ you may want to use:: rm -f err.log export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci -make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c +make C=2 CHECK=scripts/coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" ./drivers/mfd/arizona-irq.c err.log will now have the profiling information, while stdout will provide some progress information as Coccinelle moves forward with -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2] scripts: coccicheck: Correct usage of make coccicheck
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the error: ./scripts/coccicheck: line 65: -1: shift count out of range This happens because every time the C variable is specified, the shell arguments need to be "shifted" in order to take only the last argument, which is the C file to test. These shell arguments mostly comprise flags that have been set in the Makefile. However, when coccicheck is specified in the make command as a rule, the number of shell arguments is zero, thus passing the invalid value -1 to the shift command, resulting in an error. Modify coccicheck to print correct usage of make coccicheck so as to avoid the error. Signed-off-by: Sumera Priyadarsini --- Changes in v2: - Move test to only display error message --- scripts/coccicheck | 12 1 file changed, 12 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index 209bb0427b43..f3b8bf505c5f 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -61,6 +61,18 @@ COCCIINCLUDE=${COCCIINCLUDE// -include/ --include} if [ "$C" = "1" -o "$C" = "2" ]; then ONLINE=1 +if [[ $# -le 0 ]]; then + echo '' + echo 'Specifying both the variable "C" and rule "coccicheck" in the make +command results in a shift count error.' + echo '' + echo 'Try specifying "scripts/coccicheck" as a value for the CHECK variable instead.' + echo '' + echo 'Example: make C=2 CHECK=scripts/coccicheck drivers/staging/wfx/hi_t.o' + echo '' + exit 1 +fi + # Take only the last argument, which is the C file to test shift $(( $# - 1 )) OPTIONS="$COCCIINCLUDE $1" -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2] coccinelle: locks: Add balancedlock.cocci script
When acquiring locks under certain conditions, they must be released under the same conditions as well. However, sometimes, there may be missing unlocks which may lead to a potential deadlock. Add this script to detect such code segments and avoid potential deadlock situations. Signed-off-by: Sumera Priyadarsini --- Changes in v2(as suggested by Markus): - Modify usage of position variable - Modify comments - Add dependencies for rules --- scripts/coccinelle/locks/balancedlock.cocci | 164 1 file changed, 164 insertions(+) create mode 100644 scripts/coccinelle/locks/balancedlock.cocci diff --git a/scripts/coccinelle/locks/balancedlock.cocci b/scripts/coccinelle/locks/balancedlock.cocci new file mode 100644 index ..9684a9920f79 --- /dev/null +++ b/scripts/coccinelle/locks/balancedlock.cocci @@ -0,0 +1,164 @@ +/// Sometimes, locks that are acquired under certain conditions may have +/// missing unlocks leading to a potential deadlock situation. This +/// semantic patch detects such cases. +//# False positives may be generated due to locks released within a nested +//# function call or a goto block. +/// +// Confidence: Moderate +// Copyright: (C) 2020 Julia Lawall INRIA/LIP6 +// Copyright: (C) 2020 Sumera Priyadarsini + +virtual context +virtual org +virtual report + + +@prelocked@ +expression E; +position p; +@@ + +( +mutex_lock@p(E); +| +read_lock@p(E); +| +write_lock@p(E); +| +spin_lock@p(E); +| +spin_lock_bh@p(E); +| +spin_lock_irqsave@p(E, ...); +| +read_lock_irqsave@p(E, ...); +| +write_lock_irqsave@p(E, ...); +| +raw_spin_lock@p(E); +| +raw_spin_lock_irq@p(E); +| +raw_spin_lock_bh@p(E); +| +local_lock@p(E); +| +local_lock_irq@p(E); +| +local_lock_irqsave@p(E, ...); +| +read_lock_irq@p(E); +| +read_lock_bh@p(E); +| +write_lock_bh@p(E); +) + +@balanced@ depends on context || org || report@ +position prelocked.p; +position pif; +expression e,prelocked.E; +statement S1,S2; +identifier lock; +identifier unlock={mutex_unlock, + spin_unlock, + spin_unlock_bh, + spin_unlock_irqrestore, + read_unlock_irqrestore, + write_unlock_irqrestore, + raw_spin_unlock, + raw_spin_unlock_irq, + raw_spin_unlock_bh, + local_unlock, + local_unlock_irq, + local_unlock_irqrestore, + read_unlock_irq, + read_unlock_bh, + write_unlock_bh + }; +@@ + +if (e) { + ... when any +lock@p(E, ...) + ... when != E + when any +} else S1 +... when != E +when any +if@pif (e) { + ... when != E + when any + unlock(E, ...); + ... when any +} else S2 +... when != E + when any + +// + +@balanced2 depends on context || org || report@ +identifier lock, unlock = {mutex_unlock, + spin_unlock, + spin_unlock_bh, + spin_unlock_irqrestore, + read_unlock_irqrestore, + write_unlock_irqrestore, + raw_spin_unlock, + raw_spin_unlock_irq, + raw_spin_unlock_bh, + local_unlock, + local_unlock_irq, + local_unlock_irqrestore, + read_unlock_irq, + read_unlock_bh, + write_unlock_bh + }; +expression E, f, x; +statement S1, S2, S3, S4; +position prelocked.p, balanced.pif; +position j0, j1, j2, j3; +@@ + +* lock@j0@p(E, ...); +... when != E; +when != if@pif (...) S1 else S2 +when any +x@j1 = f(...); +* if (<+...x...+>) +{ + ... when != E; + when forall + when != if@pif (...) S3 else S4 +* return@j2 ...; +} +... when any +* unlock@j3(E, ...); + +// + +@script:python balanced2_org depends on org@ +j0 << balanced2.j0; +j1 << balanced2.j1; +j2 << balanced2.j2; +j3 << balanced2.j3; +@@ + +msg = "This code segment might have an unbalanced lock." +coccilib.org.print_todo(j0[0], msg) +coccilib.org.print_link(j1[0], "") +coccilib.org.print_link(j2[0], "") +coccilib.org.print_link(j3[0], "") + +// + +@script:python balanced2_report depends on report@ +j0 << balanced2.j0; +j1 << balanced2.j1; +j2 << balanced2.j2; +j3 << balanced2.j3; +@@ + +msg = "This code segment might have an unbalanced lock around lines %s,%s,%s." % (j1[0].line,j2[0].line,j3[0].line) +coccilib.report.print_report(j0[0], msg) + -- 2.25.1
[Cocci] [PATCH] scripts: coccicheck: Correct usage of make coccicheck
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the error: ./scripts/coccicheck: line 65: -1: shift count out of range This happens because every time the C variable is specified, the shell arguments need to be "shifted" in order to take only the last argument, which is the C file to test. These shell arguments mostly comprise flags that have been set in the Makefile. However, when coccicheck is specified in the make command as a rule, the number of shell arguments is zero, thus passing the invalid value -1 to the shift command, resulting in an error. Modify coccicheck to print correct usage of make coccicheck so as to avoid the error. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 12 1 file changed, 12 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index 209bb0427b43..b990c8a60a94 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -63,6 +63,18 @@ if [ "$C" = "1" -o "$C" = "2" ]; then # Take only the last argument, which is the C file to test shift $(( $# - 1 )) +err=$? +if [[ $err -ne 0 ]]; then + echo '' + echo 'Specifying both the variable "C" and rule "coccicheck" in the make +command results in a shift count error.' + echo '' + echo 'Try specifying "scripts/coccicheck" as a value for the CHECK variable instead.' + echo '' + echo 'Example: make C=2 CHECK=scripts/coccicheck drivers/staging/wfx/hi_t.o' + echo '' + exit 1 +fi OPTIONS="$COCCIINCLUDE $1" # No need to parallelize Coccinelle since this mode takes one input file. -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] coccinelle: locks: Add balancedlock.cocci script
When acquiring locks under certain conditions, they must be released under the same conditions as well. However, sometimes, there may be missing unlocks which may lead to a potential deadlock. Add this script to detect such code segments and avoid potential deadlock situations. Signed-off-by: Sumera Priyadarsini --- scripts/coccinelle/locks/balancedlock.cocci | 162 1 file changed, 162 insertions(+) create mode 100644 scripts/coccinelle/locks/balancedlock.cocci diff --git a/scripts/coccinelle/locks/balancedlock.cocci b/scripts/coccinelle/locks/balancedlock.cocci new file mode 100644 index ..fe7bc2dfeb29 --- /dev/null +++ b/scripts/coccinelle/locks/balancedlock.cocci @@ -0,0 +1,162 @@ +/// Sometimes, locks that are acquired under certain conditions may have missing unlocks +/// leading to a potential deadlock situation. This patch detects such cases. +//# False positives may be generated due to locks released within a nested +//# function call or a goto block. +/// +// Confidence: Moderate +// Copyright: (C) 2020 Julia Lawall INRIA/LIP6 + +virtual context +virtual org +virtual report + + +@prelocked@ +expression E; +position p; +@@ + +( +mutex_lock(E@p); +| +read_lock(E@p); +| +write_lock(E@p); +| +spin_lock(E@p); +| +spin_lock_bh(E@p); +| +spin_lock_irqsave(E@p, ...); +| +read_lock_irqsave(E@p, ...); +| +write_lock_irqsave(E@p, ...); +| +raw_spin_lock(E@p); +| +raw_spin_lock_irq(E@p); +| +raw_spin_lock_bh(E@p); +| +local_lock(E@p); +| +local_lock_irq(E@p); +| +local_lock_irqsave(E@p, ...); +| +read_lock_irq(E@p); +| +read_lock_bh(E@p); +| +write_lock_bh(E@p); +) + +@balanced@ +position prelocked.p; +position pif; +expression e,prelocked.E; +statement S1,S2; +identifier lock; +identifier unlock={mutex_unlock, + spin_unlock, + spin_unlock_bh, + spin_unlock_irqrestore, + read_unlock_irqrestore, + write_unlock_irqrestore, + raw_spin_unlock, + raw_spin_unlock_irq, + raw_spin_unlock_bh, + local_unlock, + local_unlock_irq, + local_unlock_irqrestore, + read_unlock_irq, + read_unlock_bh, + write_unlock_bh + }; +@@ + +if (e) { + ... when any +lock(E@p, ...) + ... when != E + when any +} else S1 +... when != E +when any +if@pif (e) { + ... when != E + when any + unlock(E, ...); + ... when any +} else S2 +... when != E + when any + +// + +@balanced2 depends on context || org || report@ +identifier lock, unlock = {mutex_unlock, + spin_unlock, + spin_unlock_bh, + spin_unlock_irqrestore, + read_unlock_irqrestore, + write_unlock_irqrestore, + raw_spin_unlock, + raw_spin_unlock_irq, + raw_spin_unlock_bh, + local_unlock, + local_unlock_irq, + local_unlock_irqrestore, + read_unlock_irq, + read_unlock_bh, + write_unlock_bh + }; +expression E, f, x; +statement S1, S2, S3, S4; +position prelocked.p, balanced.pif; +position j0, j1, j2, j3; +@@ + +* lock@j0(E@p, ...); +... when != E; +when != if@pif (...) S1 else S2 +when any +x@j1 = f(...); +* if (<+...x...+>) +{ + ... when != E; + when forall + when != if@pif (...) S3 else S4 +* return@j2 ...; +} +... when any +* unlock@j3(E, ...); + +// + +@script:python balanced2_org depends on org@ +j0 << balanced2.j0; +j1 << balanced2.j1; +j2 << balanced2.j2; +j3 << balanced2.j3; +@@ + +msg = "This code segment might have an unbalanced lock." +coccilib.org.print_todo(j0[0], msg) +coccilib.org.print_link(j1[0], "") +coccilib.org.print_link(j2[0], "") +coccilib.org.print_link(j3[0], "") + +// + +@script:python balanced2_report depends on report@ +j0 << balanced2.j0; +j1 << balanced2.j1; +j2 << balanced2.j2; +j3 << balanced2.j3; +@@ + +msg = "This code segment might have an unbalanced lock around lines %s,%s,%s." % (j1[0].line,j2[0].line,j3[0].line) +coccilib.report.print_report(j0[0], msg) + -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH V3] coccinelle: iterators: Add for_each_child.cocci script
While iterating over child nodes with the for_each functions, if control is transferred from the middle of the loop, as in the case of a break or return or goto, there is no decrement in the reference counter thus ultimately resulting in a memory leak. Add this script to detect potential memory leaks caused by the absence of of_node_put() before break, goto, or, return statements which transfer control outside the loop. Signed-off-by: Sumera Priyadarsini Changes in V2: - Add options --include-headers and --no-includes - Add 'when forall` to rules for break and goto Changes in V3: - Add return case --- .../coccinelle/iterators/for_each_child.cocci | 358 ++ 1 file changed, 358 insertions(+) create mode 100644 scripts/coccinelle/iterators/for_each_child.cocci diff --git a/scripts/coccinelle/iterators/for_each_child.cocci b/scripts/coccinelle/iterators/for_each_child.cocci new file mode 100644 index ..bc394615948e --- /dev/null +++ b/scripts/coccinelle/iterators/for_each_child.cocci @@ -0,0 +1,358 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Adds missing of_node_put() before return/break/goto statement within a for_each iterator for child nodes. +//# False positives can be due to function calls within the for_each +//# loop that may encapsulate an of_node_put. +/// +// Confidence: High +// Copyright: (C) 2020 Sumera Priyadarsini +// URL: http://coccinelle.lip6.fr +// Options: --no-includes --include-headers + +virtual patch +virtual context +virtual org +virtual report + +@r@ +local idexpression n; +expression e1,e2; +iterator name for_each_node_by_name, for_each_node_by_type, +for_each_compatible_node, for_each_matching_node, +for_each_matching_node_and_match, for_each_child_of_node, +for_each_available_child_of_node, for_each_node_with_property; +iterator i; +statement S; +expression list [n1] es; +@@ + +( +( +for_each_node_by_name(n,e1) S +| +for_each_node_by_type(n,e1) S +| +for_each_compatible_node(n,e1,e2) S +| +for_each_matching_node(n,e1) S +| +for_each_matching_node_and_match(n,e1,e2) S +| +for_each_child_of_node(e1,n) S +| +for_each_available_child_of_node(e1,n) S +| +for_each_node_with_property(n,e1) S +) +& +i(es,n,...) S +) + +@ruleone depends on patch && !context && !org && !report@ + +local idexpression r.n; +iterator r.i,i1; +expression e; +expression list [r.n1] es; +statement S; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + return n; +| + i1(...,n,...) S +| +- return of_node_get(n); ++ return n; +| ++ of_node_put(n); +? return ...; +) + ... when any + } + +@ruletwo depends on patch && !context && !org && !report@ + +local idexpression r.n; +iterator r.i,i1,i2; +expression e,e1; +expression list [r.n1] es; +statement S,S2; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| ++ of_node_put(n); +? break; +) + ... when any + } +... when != n +when strict +when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +@rulethree depends on patch && !context && !org && !report exists@ + +local idexpression r.n; +iterator r.i,i1,i2; +expression e,e1; +identifier l; +expression list [r.n1] es; +statement S,S2; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| ++ of_node_put(n); +? goto l; +) + ... when any + } +... when exists +l: ... when != n + when strict + when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +// + +@ruleone_context depends on !patch && (context || org || report) exists@ +statement S; +expression e; +expression list[r.n1] es; +iterator r.i, i1; +local idexpression r.n; +position j0, j1; +@@ + + i@j0(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + return n; +| + i1(...,n,...) S +| + return @j1 ...; +) + ... when any + } + +@ruleone_disj depends on !patch && (context || org || report)@ +expression list[r.n1] es; +iterator r.i; +local idexpression r.n; +position ruleone_context.j0, ruleone_context.j1; +@@ + +* i@j0(es,n,...) { + ... +*return @j1...; + ... when any + } + +@ruletwo_context depends on !patch && (context || org || report) exists@ +statement S, S2; +expression e, e1; +expression list[r.n1] es; +iterator r.i, i1, i2; +local idexpression r.n; +position j0, j2; +@@ + + i@j0(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| + break@j2; +) + ... when any + } +... when != n +when strict +when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +@ruletwo_disj depends on !patch && (context || org || report)@ +statement S2; +expression e1; +expression list[r.n1] es; +iterator r.i, i2; +local idexpression r.n; +position ruletwo_context.j0, ruletwo_context.j2; +@@ + +* i@j0(es,n,...) { + ... +*break @j2; + ... when any + } +... when != n +when strict
[Cocci] [PATCH V2] coccinelle: iterators: Add for_each_child.cocci script
While iterating over child nodes with the for_each functions, if control is transferred from the middle of the loop, as in the case of a break or return or goto, there is no decrement in the reference counter thus ultimately resulting in a memory leak. Add this script to detect potential memory leaks caused by the absence of of_node_put() before break, goto, or, return statements which transfer control outside the loop. Signed-off-by: Sumera Priyadarsini Changes in V2: - Add options --include-headers and --no-includes - Add 'when forall` to rules for break and goto --- .../coccinelle/iterators/for_each_child.cocci | 355 ++ 1 file changed, 355 insertions(+) create mode 100644 scripts/coccinelle/iterators/for_each_child.cocci diff --git a/scripts/coccinelle/iterators/for_each_child.cocci b/scripts/coccinelle/iterators/for_each_child.cocci new file mode 100644 index ..57caf9b8d7a5 --- /dev/null +++ b/scripts/coccinelle/iterators/for_each_child.cocci @@ -0,0 +1,355 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Adds missing of_node_put() before return/break/goto statement within a for_each iterator for child nodes. +//# False positives can be due to function calls within the for_each +//# loop that may encapsulate an of_node_put. +/// +// Confidence: High +// Copyright: (C) 2020 Sumera Priyadarsini +// URL: http://coccinelle.lip6.fr +// Options: --no-includes --include-headers + +virtual patch +virtual context +virtual org +virtual report + +@r@ +local idexpression n; +expression e1,e2; +iterator name for_each_node_by_name, for_each_node_by_type, +for_each_compatible_node, for_each_matching_node, +for_each_matching_node_and_match, for_each_child_of_node, +for_each_available_child_of_node, for_each_node_with_property; +iterator i; +statement S; +expression list [n1] es; +@@ + +( +( +for_each_node_by_name(n,e1) S +| +for_each_node_by_type(n,e1) S +| +for_each_compatible_node(n,e1,e2) S +| +for_each_matching_node(n,e1) S +| +for_each_matching_node_and_match(n,e1,e2) S +| +for_each_child_of_node(e1,n) S +| +for_each_available_child_of_node(e1,n) S +| +for_each_node_with_property(n,e1) S +) +& +i(es,n,...) S +) + +@ruleone depends on patch && !context && !org && !report@ + +local idexpression r.n; +iterator r.i,i1; +expression e; +expression list [r.n1] es; +statement S; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + return n; +| + i1(...,n,...) S +| ++ of_node_put(n); +? return ...; +) + ... when any + } + +@ruletwo depends on patch && !context && !org && !report@ + +local idexpression r.n; +iterator r.i,i1,i2; +expression e,e1; +expression list [r.n1] es; +statement S,S2; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| ++ of_node_put(n); +? break; +) + ... when any + } +... when != n +when strict +when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +@rulethree depends on patch && !context && !org && !report exists@ + +local idexpression r.n; +iterator r.i,i1,i2; +expression e,e1; +identifier l; +expression list [r.n1] es; +statement S,S2; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| ++ of_node_put(n); +? goto l; +) + ... when any + } +... when exists +l: ... when != n + when strict + when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +// + +@ruleone_context depends on !patch && (context || org || report) exists@ +statement S; +expression e; +expression list[r.n1] es; +iterator r.i, i1; +local idexpression r.n; +position j0, j1; +@@ + + i@j0(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + return n; +| + i1(...,n,...) S +| + return @j1 ...; +) + ... when any + } + +@ruleone_disj depends on !patch && (context || org || report)@ +expression list[r.n1] es; +iterator r.i; +local idexpression r.n; +position ruleone_context.j0, ruleone_context.j1; +@@ + +* i@j0(es,n,...) { + ... +*return @j1...; + ... when any + } + +@ruletwo_context depends on !patch && (context || org || report) exists@ +statement S, S2; +expression e, e1; +expression list[r.n1] es; +iterator r.i, i1, i2; +local idexpression r.n; +position j0, j2; +@@ + + i@j0(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| + break@j2; +) + ... when any + } +... when != n +when strict +when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +@ruletwo_disj depends on !patch && (context || org || report)@ +statement S2; +expression e1; +expression list[r.n1] es; +iterator r.i, i2; +local idexpression r.n; +position ruletwo_context.j0, ruletwo_context.j2; +@@ + +* i@j0(es,n,...) { + ... +*break @j2; + ... when any + } +... when != n +when strict +when forall +( + n = e1; +| +?i2(...,n,...) S2 +) + +@rulethree_conte
[Cocci] [PATCH v4 3/3] Documentation: Coccinelle: Modify Parallelisation information in docs
This patchset modifies coccicheck to use at most one thread per core by default in machines with more than 4 hyperthreads for optimal performance. Modify documentation in coccinelle.rst to reflect the same. Signed-off-by: Sumera Priyadarsini --- Documentation/dev-tools/coccinelle.rst | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 74c5e6aeeff5..530d8d313601 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -130,8 +130,10 @@ To enable verbose messages set the V= variable, for example:: Coccinelle parallelization -- -By default, coccicheck tries to run as parallel as possible. To change -the parallelism, set the J= variable. For example, to run across 4 CPUs:: +By default, coccicheck uses at most 1 thread per core in a machine +with more than 4 hyperthreads. In a machine with upto 4 threads, +all threads are used. To change the parallelism, set the J= variable. +For example, to run across 4 CPUs:: make coccicheck MODE=report J=4 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v4 2/3] scripts: coccicheck: Change default condition for parallelism
Currently, Coccinelle uses at most one thread per core by default in machines with more than 2 hyperthreads. However, for systems with only 4 hyperthreads, this does not improve performance. Modify coccicheck to use all available threads in machines with upto 4 hyperthreads. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index d67907b8a38b..209bb0427b43 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -79,7 +79,7 @@ else THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd "[:digit:]") if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) - if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 4 ] ; then NPROC=$((NPROC/2)) fi else -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v4 1/3] scripts: coccicheck: Add quotes to improve portability
While fetching the number of threads per core with lscpu, the [:digit:] set is used for translation of digits from 0-9. However, using [:digit:] instead of "[:digit:]" does not seem to work uniformly for some shell types and configurations (such as zsh). Therefore, modify coccicheck to use double quotes around the [:digit:] set for uniformity and better portability. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index 6789751607f5..d67907b8a38b 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -76,7 +76,7 @@ else fi # Use only one thread per core by default if hyperthreading is enabled -THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd [:digit:]) +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd "[:digit:]") if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v4 0/3] Improve Coccinelle Parallelisation
Presently, Coccinelle uses at most one thread per core to improve performance in machines with more than 2 hyperthreads. Modify coccicheck to use all available threads in machines upto 4 hyperthreads. Further, modify the coccicheck script to improve portability. Modify documentation to reflect the same. Sumera Priyadarsini (3): scripts: coccicheck: Add quotes to improve portability scripts: coccicheck: Change default condition for parallelism Documentation: Coccinelle: Modify Parallelisation information in docs Documentation/dev-tools/coccinelle.rst | 6 -- scripts/coccicheck | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 2/2 V3] Documentation: Coccinelle: Modify parallelisation information in docs
This patchset modifies coccicheck to use at most one thread per core by default in machines with more than 4 hyperthreads for optimal performance. Modify documentation in coccinelle.rst to reflect the same. Signed-off-by: Sumera Priyadarsini --- Changes in V2: Update scripts/coccicheck to use all available threads in machines with upto 4 hyperthreads. --- Documentation/dev-tools/coccinelle.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 74c5e6aeeff5..6fdc462689d5 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -130,8 +130,9 @@ To enable verbose messages set the V= variable, for example:: Coccinelle parallelization -- -By default, coccicheck tries to run as parallel as possible. To change -the parallelism, set the J= variable. For example, to run across 4 CPUs:: +By default, coccicheck uses at most 1 thread per core in a machine with more +than 4 hyperthreads. In a machine with upto 4 threads, all threads are used. +To change the parallelism, set the J= variable. For example, to run across 4 CPUs:: make coccicheck MODE=report J=4 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 1/2 V3] scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement parallelisation. However, when all available threads are used, a decrease in performance is noted. The elapsed time is minimum when at most one thread per core is used. For example, on benchmarking the semantic patch kfree.cocci for usb/serial using hyperfine, the outputs obtained for J=5 and J=2 are 1.32 and 1.90 times faster than those for J=10 and J=9 respectively for two separate runs. For the larger drivers/staging directory, minimium elapsed time is obtained for J=3 which is 1.86 times faster than that for J=12. The optimal J value does not exceed 6 in any of the test runs. The benchmarks are run on a machine with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all. To improve performance, modify coccicheck to use at most only one thread per core by default in machines with more than 4 hyperthreads. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Change commit message as suggested by Julia Lawall Changes in V3: - Use J/2 as optimal value for machines with more than 8 hyperthreads as well. Changes in V4: - Use J as optimal value for machines with less than or equal to 4 hyperthreads. --- scripts/coccicheck | 5 + 1 file changed, 5 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..bafc55141a73 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -75,8 +75,13 @@ else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi +# Use only one thread per core by default if hyperthreading is enabled +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd "[:digit:]") if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 4 ] ; then + NPROC=$((NPROC/2)) + fi else NPROC="$J" fi -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 0/2 V3] Improve Coccinelle Parallelisation
Coccinelle utilises all available threads to implement parallelisation. However, this results in a decrease in performance. This patchset aims to improve performance by modifying cocciccheck to use at most one thread per core by default in machines with more than 4 hyperthreads. Sumera Priyadarsini (2): scripts: coccicheck: Change default value for parallelism Documentation: Coccinelle: Modify parallelisation information in docs Documentation/dev-tools/coccinelle.rst | 5 +++-- scripts/coccicheck | 5 + 2 files changed, 8 insertions(+), 2 deletions(-) -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 2/2 V2] Documentation: Coccinelle: Modify parallelisation information in docs
This patchset modifies coccicheck to use at most one thread per core by default for optimal performance. Modify documentation in coccinelle.rst to reflect the same. Signed-off-by: Sumera Priyadarsini --- Documentation/dev-tools/coccinelle.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 74c5e6aeeff5..a27a4867018c 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -130,8 +130,8 @@ To enable verbose messages set the V= variable, for example:: Coccinelle parallelization -- -By default, coccicheck tries to run as parallel as possible. To change -the parallelism, set the J= variable. For example, to run across 4 CPUs:: +By default, coccicheck uses at most only one thread per core of the system. +To change the parallelism, set the J= variable. For example, to run across 4 CPUs:: make coccicheck MODE=report J=4 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 0/2 v2] Improve Coccinelle Parallelisation
Coccinelle utilises all available threads to implement parallelisation. However, this results in a decrease in performance. This patchset aims to improve performance by modifying cocciccheck to use at most one thread per core by default. Sumera Priyadarsini (2): scripts: coccicheck: Change default value for parallelism Documentation: Coccinelle: Modify parallelisation information in docs Documentation/dev-tools/coccinelle.rst | 4 ++-- scripts/coccicheck | 5 + 2 files changed, 7 insertions(+), 2 deletions(-) -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 1/2 V2] scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement parallelisation. However, when all available threads are used, a decrease in performance is noted. The elapsed time is minimum when at most one thread per core is used. For example, on benchmarking the semantic patch kfree.cocci for usb/serial using hyperfine, the outputs obtained for J=5 and J=2 are 1.32 and 1.90 times faster than those for J=10 and J=9 respectively for two separate runs. For the larger drivers/staging directory, minimium elapsed time is obtained for J=3 which is 1.86 times faster than that for J=12. The optimal J value does not exceed 6 in any of the test runs. The benchmarks are run on a machine with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all. To improve performance, modify coccicheck to use at most only one thread per core by default. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Change commit message as suggested by Julia Lawall Changes in V3: - Use J/2 as optimal value for machines with more than 8 hyperthreads as well. --- scripts/coccicheck | 5 + 1 file changed, 5 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..a72aa6c037ff 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -75,8 +75,13 @@ else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi +# Use only one thread per core by default if hyperthreading is enabled +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd [:digit:]) if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then + NPROC=$((NPROC/2)) + fi else NPROC="$J" fi -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] scripts: coccicheck: Refactor display messages on coccinelle start up
Currently, coccinelle starts by printing "When using "patch" mode, carefully review the patch before submitting it." Modify coccicheck to print this message only when the user has explicitly selected "patch" or "chain" mode. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..07d1b5831bf6 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -118,7 +118,9 @@ fi if [ "$ONLINE" = "0" ] ; then echo '' echo 'Please check for false positives in the output before submitting a patch.' -echo 'When using "patch" mode, carefully review the patch before submitting it.' +if [ "$MODE" = "patch" -o "$MODE" = "chain" ] ; then +echo 'When using "patch" mode, carefully review the patch before submitting it.' +fi echo '' fi -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 2/2] Documentation: Coccinelle: Modify parallelisation information in docs
This patchset modifies coccicheck to use at most one thread per core by default for optimal performance. Modify documentation in coccinelle.rst to reflect the same. Signed-off-by: Sumera Priyadarsini --- Documentation/dev-tools/coccinelle.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 74c5e6aeeff5..a27a4867018c 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -130,8 +130,8 @@ To enable verbose messages set the V= variable, for example:: Coccinelle parallelization -- -By default, coccicheck tries to run as parallel as possible. To change -the parallelism, set the J= variable. For example, to run across 4 CPUs:: +By default, coccicheck uses at most only one thread per core of the system. +To change the parallelism, set the J= variable. For example, to run across 4 CPUs:: make coccicheck MODE=report J=4 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 1/2] scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement parallelisation. However, when all available threads are used, a decrease in performance is noted. The elapsed time is minimum when at most one thread per core is used. For example, on benchmarking the semantic patch kfree.cocci for usb/serial using hyperfine, the outputs obtained for J=5 and J=2 are 1.32 and 1.90 times faster than those for J=10 and J=9 respectively for two separate runs. For the larger drivers/staging directory, minimium elapsed time is obtained for J=3 which is 1.86 times faster than that for J=12. The optimal J value does not exceed 6 in any of the test runs. The benchmarks are run on a machine with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all. To improve performance, modify coccicheck to use at most only one thread per core by default. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Change commit message as suggested by Julia Lawall Changes in V3: - Use J/2 as optimal value for machines with more than 8 hyperthreads as well. --- scripts/coccicheck | 5 + 1 file changed, 5 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..a72aa6c037ff 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -75,8 +75,13 @@ else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi +# Use only one thread per core by default if hyperthreading is enabled +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd [:digit:]) if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then + NPROC=$((NPROC/2)) + fi else NPROC="$J" fi -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH 0/2] Improve Coccinelle Parallelisation
Coccinelle utilises all available threads to implement parallelisation. However, this results in a decrease in performance. This patchset aims to improve performance by modifying cocciccheck to use at most one thread per core by default. Sumera Priyadarsini (2): scripts: coccicheck: Change default value for parallelism Documentation: Coccinelle: Modify parallelisation information in docs Documentation/dev-tools/coccinelle.rst | 4 ++-- scripts/coccicheck | 5 + 2 files changed, 7 insertions(+), 2 deletions(-) -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] coccinelle: iterators: Add for_each_child.cocci script
While iterating over child nodes with the for_each functions, if control is transferred from the middle of the loop, as in the case of a break or return or goto, there is no decrement in the reference counter thus ultimately resulting in a memory leak. Add this script to detect potential memory leaks caused by the absence of of_node_put() before break, goto, or, return statements which transfer control outside the loop. Signed-off-by: Sumera Priyadarsini --- .../coccinelle/iterators/for_each_child.cocci | 348 ++ 1 file changed, 348 insertions(+) create mode 100644 scripts/coccinelle/iterators/for_each_child.cocci diff --git a/scripts/coccinelle/iterators/for_each_child.cocci b/scripts/coccinelle/iterators/for_each_child.cocci new file mode 100644 index ..0abc12ca2ad3 --- /dev/null +++ b/scripts/coccinelle/iterators/for_each_child.cocci @@ -0,0 +1,348 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Adds missing of_node_put() before return/break/goto statement within a for_each iterator for child nodes. +//# False positives can be due to function calls within the for_each +//# loop that may encapsulate an of_node_put. +/// +// Confidence: High +// Copyright: (C) 2020 Sumera Priyadarsini +// URL: http://coccinelle.lip6.fr + +virtual patch +virtual context +virtual org +virtual report + +@r@ +local idexpression n; +expression e1,e2; +iterator name for_each_node_by_name, for_each_node_by_type, +for_each_compatible_node, for_each_matching_node, +for_each_matching_node_and_match, for_each_child_of_node, +for_each_available_child_of_node, for_each_node_with_property; +iterator i; +statement S; +expression list [n1] es; +@@ + +( +( +for_each_node_by_name(n,e1) S +| +for_each_node_by_type(n,e1) S +| +for_each_compatible_node(n,e1,e2) S +| +for_each_matching_node(n,e1) S +| +for_each_matching_node_and_match(n,e1,e2) S +| +for_each_child_of_node(e1,n) S +| +for_each_available_child_of_node(e1,n) S +| +for_each_node_with_property(n,e1) S +) +& +i(es,n,...) S +) + +@ruleone depends on patch && !context && !org && !report@ + +local idexpression r.n; +iterator r.i,i1; +expression e; +expression list [r.n1] es; +statement S; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + return n; +| + i1(...,n,...) S +| ++ of_node_put(n); +? return ...; +) + ... when any + } + +@ruletwo depends on patch && !context && !org && !report@ + +local idexpression r.n; +iterator r.i,i1,i2; +expression e,e1; +expression list [r.n1] es; +statement S,S2; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| ++ of_node_put(n); +? break; +) + ... when any + } +... when != n +when strict +( + n = e1; +| +?i2(...,n,...) S2 +) + +@rulethree depends on patch && !context && !org && !report exists@ + +local idexpression r.n; +iterator r.i,i1,i2; +expression e,e1; +identifier l; +expression list [r.n1] es; +statement S,S2; +@@ + + i(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| ++ of_node_put(n); +? goto l; +) + ... when any + } +... when exists +l: ... when != n + when strict +( + n = e1; +| +?i2(...,n,...) S2 +) + +// + +@ruleone_context depends on !patch && (context || org || report) exists@ +statement S; +expression e; +expression list[r.n1] es; +iterator r.i, i1; +local idexpression r.n; +position j0, j1; +@@ + + i@j0(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + return n; +| + i1(...,n,...) S +| + return @j1 ...; +) + ... when any + } + +@ruleone_disj depends on !patch && (context || org || report)@ +expression list[r.n1] es; +iterator r.i; +local idexpression r.n; +position ruleone_context.j0, ruleone_context.j1; +@@ + +* i@j0(es,n,...) { + ... +*return @j1...; + ... when any + } + +@ruletwo_context depends on !patch && (context || org || report) exists@ +statement S, S2; +expression e, e1; +expression list[r.n1] es; +iterator r.i, i1, i2; +local idexpression r.n; +position j0, j2; +@@ + + i@j0(es,n,...) { + ... +( + of_node_put(n); +| + e = n +| + i1(...,n,...) S +| + break@j2; +) + ... when any + } +... when != n +when strict +( + n = e1; +| +?i2(...,n,...) S2 +) + +@ruletwo_disj depends on !patch && (context || org || report)@ +statement S2; +expression e1; +expression list[r.n1] es; +iterator r.i, i2; +local idexpression r.n; +position ruletwo_context.j0, ruletwo_context.j2; +@@ + +* i@j0(es,n,...) { + ... +*break @j2; + ... when any + } +... when != n +when strict +( + n = e1; +| +?i2(...,n,...) S2 +) + +@rulethree_context depends on !patch && (context || org || report) exists@ +identifier l; +statement S,S2; +expression e, e1; +expression list[r.n1] es; +iterator r.i, i1, i2; +local idexpression r.n; +position j0, j3; +@@ + + i@j0(es,n,...) {
[Cocci] [RFC PATCH] scripts: coccicheck: Improve error feedback when coccicheck fails
Currently, coccicheck fails with only the message "coccicheck failed" and the error code for the failure. To obtain the error logs, one needs to specify a debug file using the DEBUG_FILE option. Modify coccicheck to display error logs when it crashes unless DEBUG_FILE is set, in which case, the error logs are stored in the specified debug file. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..dbeafa21f359 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -126,8 +126,14 @@ run_cmd_parmap() { if [ $VERBOSE -ne 0 ] ; then echo "Running ($NPROC in parallel): $@" fi - echo $@ >>$DEBUG_FILE - $@ 2>>$DEBUG_FILE + if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then +echo $@>>$DEBUG_FILE +$@ 2>>$DEBUG_FILE +else +echo $@ +$@ 2>&1 + fi + err=$? if [[ $err -ne 0 ]]; then echo "coccicheck failed" -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH V3] scripts: coccicheck: Do not use shift command when rule is specified
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the error: ./scripts/coccicheck: line 65: -1: shift count out of range This happens because every time the C variable is specified, the shell arguments need to be "shifted" in order to take only the last argument, which is the C file to test. These shell arguments mostly comprise flags that have been set in the Makefile. However, when coccicheck is specified in the make command as a rule, the number of shell arguments is zero, thus passing the invalid value -1 to the shift command, resulting in an error. Modify coccicheck to use the shift command only when number of shell arguments is not zero. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Fix spelling errors as suggested by Markus Elfring --- scripts/coccicheck | 16 +--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..5c8df337e1e3 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -61,9 +61,19 @@ COCCIINCLUDE=${COCCIINCLUDE// -include/ --include} if [ "$C" = "1" -o "$C" = "2" ]; then ONLINE=1 -# Take only the last argument, which is the C file to test -shift $(( $# - 1 )) -OPTIONS="$COCCIINCLUDE $1" +# If the rule coccicheck is specified when calling make, number of +# arguments is zero +if [ $# -ne 0 ]; then + # Take only the last argument, which is the C file to test + shift $(( $# -1 )) + OPTIONS="$COCCIINCLUDE $1" +else + if [ "$KBUILD_EXTMOD" = "" ] ; then + OPTIONS="--dir $srctree $COCCIINCLUDE" + else + OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" + fi +fi # No need to parallelize Coccinelle since this mode takes one input file. NPROC=1 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH V2] scripts: coccicheck: Do not use shift command when rule is specfified
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the error: ./scripts/coccicheck: line 65: -1: shift count out of range This happens because every time the C variable is specified, the shell arguments need to be "shifted" in order to take only the last argument, which is the C file to test. These shell arguments mostly comprise flags that have been set in the Makefile. However, when coccicheck is specified in the make command as a rule, the number of shell arguments is zero, thus passing the invalid value -1 to the shift command, resulting in an error. Modify coccicheck to use the shift command only when number of shell arguments is not zero. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Fix spelling errors as suggested by Markus Elfring --- scripts/coccicheck | 16 +--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..5c8df337e1e3 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -61,9 +61,19 @@ COCCIINCLUDE=${COCCIINCLUDE// -include/ --include} if [ "$C" = "1" -o "$C" = "2" ]; then ONLINE=1 -# Take only the last argument, which is the C file to test -shift $(( $# - 1 )) -OPTIONS="$COCCIINCLUDE $1" +# If the rule coccicheck is specified when calling make, number of +# arguments is zero +if [ $# -ne 0 ]; then + # Take only the last argument, which is the C file to test + shift $(( $# -1 )) + OPTIONS="$COCCIINCLUDE $1" +else + if [ "$KBUILD_EXTMOD" = "" ] ; then + OPTIONS="--dir $srctree $COCCIINCLUDE" + else + OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" + fi +fi # No need to parallelize Coccinelle since this mode takes one input file. NPROC=1 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH] scripts: coccicheck: Do not use shift command when rule is specified
On Wed, Sep 09, 2020 at 08:52:19AM +0200, Markus Elfring wrote: > I find it helpful to avoid typos (like the following) in the change > description. > > > > … Makfeile. … > > … Makefile. … > > > > … paasing … > > … passing … > > > > …, resuting … > > …, resulting … > > > > This patch modifies coccicheck … > I did make those errors but I also rectified them. This is strange because my commit message shows the rectified version. Either way, I will send a v2. Thanks for pointing this out. > Would an imperative wording be preferred for the commit message? > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?id=34d4ddd359dbcdf6c5fb3f85a179243d7a1cb7f8#n151 > > Regards, > Markus ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] scripts: coccicheck: Do not use shift command when rule is specfified
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the error: ./scripts/coccicheck: line 65: -1: shift count out of range This happens because every time the C variable is specified, the shell arguments need to be "shifted" in order to take only the last argument, which is the C file to test. These shell arguments mostly comprise flags that have been set in the Makfeile. However, when coccicheck is specified in the make command as a rule, number of shell arguments is zero, thus paasing the invalid value -1 to the shift command, resuting in an error. This patch modifies coccicheck to use the shift command only when number of shell arguments is not zero. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 16 +--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..5c8df337e1e3 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -61,9 +61,19 @@ COCCIINCLUDE=${COCCIINCLUDE// -include/ --include} if [ "$C" = "1" -o "$C" = "2" ]; then ONLINE=1 -# Take only the last argument, which is the C file to test -shift $(( $# - 1 )) -OPTIONS="$COCCIINCLUDE $1" +# If the rule coccicheck is specified when calling make, number of +# arguments is zero +if [ $# -ne 0 ]; then + # Take only the last argument, which is the C file to test + shift $(( $# -1 )) + OPTIONS="$COCCIINCLUDE $1" +else + if [ "$KBUILD_EXTMOD" = "" ] ; then + OPTIONS="--dir $srctree $COCCIINCLUDE" + else + OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" + fi +fi # No need to parallelize Coccinelle since this mode takes one input file. NPROC=1 -- 2.25.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH V3] scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement parallelisation. However, when all available threads are used, a decrease in performance is noted. The elapsed time is minimum when at most one thread per core is used. For example, on benchmarking the semantic patch kfree.cocci for usb/serial using hyperfine, the outputs obtained for J=5 and J=2 are 1.32 and 1.90 times faster than those for J=10 and J=9 respectively for two separate runs. For the larger drivers/staging directory, minimium elapsed time is obtained for J=3 which is 1.86 times faster than that for J=12. The optimal J value does not exceed 6 in any of the test runs. The benchmarks are run on a machine with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all. To improve performance, modify coccicheck to use at most only one thread per core by default. Signed-off-by: Sumera Priyadarsini --- Changes in V2: - Change commit message as suggested by Julia Lawall Changes in V3: - Use J/2 as optimal value for machines with more than 8 hyperthreads as well. --- scripts/coccicheck | 5 + 1 file changed, 5 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..a72aa6c037ff 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -75,8 +75,13 @@ else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi +# Use only one thread per core by default if hyperthreading is enabled +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd [:digit:]) if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then + NPROC=$((NPROC/2)) + fi else NPROC="$J" fi -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH V2] scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement parallelisation. However, when all available threads are used, a decrease in performance is noted. The elapsed time is minimum when at most one thread per core is used. For example, on benchmarking the semantic patch kfree.cocci for usb/serial using hyperfine, the outputs obtained for J=5 and J=2 are 1.32 and 1.90 times faster than those for J=10 and J=9 respectively for two separate runs. For the larger drivers/staging directory, minimium elapsed time is obtained for J=3 which is 1.86 times faster than that for J=12. The optimal J value does not exceed 6 in any of the test runs. The benchmarks are run on a machine with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all. To improve performance, modify coccicheck to use at most only one thread per core by default. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 9 + 1 file changed, 9 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..dd228dcc915e 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -75,8 +75,17 @@ else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi +# Use only one thread per core by default if hyperthreading is enabled +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd [:digit:]) if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then + if [ $NPROC -gt 8 ] ; then + NPROC=$((NPROC/4)) + else + NPROC=$((NPROC/2)) + fi + fi else NPROC="$J" fi -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH] scripts: coccicheck: Change default value for parallelism
On Thu, 13 Aug, 2020, 1:45 AM Julia Lawall, wrote: > > > On Wed, 12 Aug 2020, Sumera Priyadarsini wrote: > > > By default, coccicheck utilizes all available threads to implement > > parallelisation. However, when hyperthreading is enabled, this leads > > to all threads per core being occupied resulting in longer wall-clock > > times and higher power consumption. > > I have the feeling that the above sentence is not quite optimal. > Actually, using all of the available hardware threads would not be a bad > thing, if it was giving a benefit. The point is that it doesn't. It > makes the performance worse instead. > How does this sound? However, when all available threads are used, a decrease in performance was noted. The elapsed time was minimum when at most one thread per core was used. > > Hence, to improve performance, > > modify coccicheck to use only one thread per core atmost. > > "atmost" is not a word. It would be clearer to say "to use at most one > thread per core". > > > In the cases where the total number of threads is more than 8 and > > hyperthreading is enabled, it was observed that optimum performance > > is achieved around one-fourth of the total number of cores. > > Modify the script further to accommodate this use case. > > It would be nice to give some performance numbers and some information > about the machine used. > Alright, will add this. > thanks, > julia > > > > > Signed-off-by: Sumera Priyadarsini > > --- > > scripts/coccicheck | 9 + > > 1 file changed, 9 insertions(+) > > > > diff --git a/scripts/coccicheck b/scripts/coccicheck > > index e04d328210ac..dd228dcc915e 100755 > > --- a/scripts/coccicheck > > +++ b/scripts/coccicheck > > @@ -75,8 +75,17 @@ else > > OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" > > fi > > > > +# Use only one thread per core by default if hyperthreading is > enabled > > +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd > [:digit:]) > > if [ -z "$J" ]; then > > NPROC=$(getconf _NPROCESSORS_ONLN) > > + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then > > + if [ $NPROC -gt 8 ] ; then > > + NPROC=$((NPROC/4)) > > + else > > + NPROC=$((NPROC/2)) > > + fi > > + fi > > else > > NPROC="$J" > > fi > > -- > > 2.17.1 > > > > ___ > > Cocci mailing list > > Cocci@systeme.lip6.fr > > https://systeme.lip6.fr/mailman/listinfo/cocci > > > ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] scripts: coccicheck: Change default value for parallelism
By default, coccicheck utilizes all available threads to implement parallelisation. However, when hyperthreading is enabled, this leads to all threads per core being occupied resulting in longer wall-clock times and higher power consumption. Hence, to improve performance, modify coccicheck to use only one thread per core atmost. In the cases where the total number of threads is more than 8 and hyperthreading is enabled, it was observed that optimum performance is achieved around one-fourth of the total number of cores. Modify the script further to accommodate this use case. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 9 + 1 file changed, 9 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..dd228dcc915e 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -75,8 +75,17 @@ else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi +# Use only one thread per core by default if hyperthreading is enabled +THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd [:digit:]) if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) + if [ $THREADS_PER_CORE -gt 1 -a $NPROC -gt 2 ] ; then + if [ $NPROC -gt 8 ] ; then + NPROC=$((NPROC/4)) + else + NPROC=$((NPROC/2)) + fi + fi else NPROC="$J" fi -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v4] documentation: coccinelle: Improve command example for make C={1, 2}
Modify coccinelle documentation to further clarify the usage of the makefile C variable by coccicheck. Signed-off-by: Sumera Priyadarsini --- Changes in v4: - Modify commit message to clarify C is a variable --- Documentation/dev-tools/coccinelle.rst | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 6c791af1c859..74c5e6aeeff5 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -175,13 +175,20 @@ For example, to check drivers/net/wireless/ one may write:: make coccicheck M=drivers/net/wireless/ To apply Coccinelle on a file basis, instead of a directory basis, the -following command may be used:: +C variable is used by the makefile to select which files to work with. +This variable can be used to run scripts for the entire kernel, a +specific directory, or for a single file. -make C=1 CHECK="scripts/coccicheck" +For example, to check drivers/bluetooth/bfusb.c, the value 1 is +passed to the C variable to check files that make considers +need to be compiled.:: -To check only newly edited code, use the value 2 for the C flag, i.e.:: +make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o -make C=2 CHECK="scripts/coccicheck" +The value 2 is passed to the C variable to check files regardless of +whether they need to be compiled or not.:: + +make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o In these modes, which work on a file basis, there is no information about semantic patches displayed, and no commit message proposed. -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v3] documentation: coccinelle: Improve command example for make C={1, 2}
Modify coccinelle documentation to further clarify the usage of the makefile C variable flag by coccicheck. Signed-off-by: Sumera Priyadarsini --- Changes in v3: - Remove quotes as suggested by Markus Elfring - Change in wording, and punctuation, as suggested by Julia Lawall --- Documentation/dev-tools/coccinelle.rst | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 6c791af1c859..74c5e6aeeff5 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -175,13 +175,20 @@ For example, to check drivers/net/wireless/ one may write:: make coccicheck M=drivers/net/wireless/ To apply Coccinelle on a file basis, instead of a directory basis, the -following command may be used:: +C variable is used by the makefile to select which files to work with. +This variable can be used to run scripts for the entire kernel, a +specific directory, or for a single file. -make C=1 CHECK="scripts/coccicheck" +For example, to check drivers/bluetooth/bfusb.c, the value 1 is +passed to the C variable to check files that make considers +need to be compiled.:: -To check only newly edited code, use the value 2 for the C flag, i.e.:: +make C=1 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o -make C=2 CHECK="scripts/coccicheck" +The value 2 is passed to the C variable to check files regardless of +whether they need to be compiled or not.:: + +make C=2 CHECK=scripts/coccicheck drivers/bluetooth/bfusb.o In these modes, which work on a file basis, there is no information about semantic patches displayed, and no commit message proposed. -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2] documentation: coccinelle: Improve command example for make C={1, 2}
Modify coccinelle documentation to further clarify the usage of the makefile C variable flag by coccicheck. Signed-off-by: Sumera Priyadarsini --- Changes in v2: - Change the message tone to imperative as suggested by Markus Elfring - Add examples for using a specific file and explain in detail the usage of the C variable, as suggested by Julia Lawall --- Documentation/dev-tools/coccinelle.rst | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 6c791af1c859..bbcb4f7e8b5c 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -175,13 +175,20 @@ For example, to check drivers/net/wireless/ one may write:: make coccicheck M=drivers/net/wireless/ To apply Coccinelle on a file basis, instead of a directory basis, the -following command may be used:: +C flag is used. The C flag is a variable used by the makefile +to select which files to work with. This flag can be used to +run scripts for the entire kernel, a specific directory, +or for a single file. For example, to check drivers/bluetooth/bfusb.c, -make C=1 CHECK="scripts/coccicheck" +The value 1 is passed to the C flag to check for files that make considers +need to be recompiled.:: -To check only newly edited code, use the value 2 for the C flag, i.e.:: +make C=1 CHECK="scripts/coccicheck" "drivers/bluetooth/bfusb.o" -make C=2 CHECK="scripts/coccicheck" +The value 2 is passed to the C flag to check for files regardless of +whether they need to be recompiled or not.:: + +make C=2 CHECK="scripts/coccicheck" "drivers/bluetooth/bfusb.o" In these modes, which work on a file basis, there is no information about semantic patches displayed, and no commit message proposed. -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] documentation: coccinelle: Improve command example
This patch modifies the coccinelle documentation to add further description for the usage of the C variable flags by coccicheck. Signed-off-by: Sumera Priyadarsini --- Documentation/dev-tools/coccinelle.rst | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Documentation/dev-tools/coccinelle.rst b/Documentation/dev-tools/coccinelle.rst index 6c791af1c859..dfc5d390307b 100644 --- a/Documentation/dev-tools/coccinelle.rst +++ b/Documentation/dev-tools/coccinelle.rst @@ -177,9 +177,12 @@ For example, to check drivers/net/wireless/ one may write:: To apply Coccinelle on a file basis, instead of a directory basis, the following command may be used:: +To check only recompiled files, use the value 1 for the C flag, i.e.:: + make C=1 CHECK="scripts/coccicheck" -To check only newly edited code, use the value 2 for the C flag, i.e.:: +To check sourcefiles regardless of whether they are recompiled or not, +use the value 2 for the C flag, i.e.:: make C=2 CHECK="scripts/coccicheck" -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH v2] scripts: coccicheck: Add chain mode to list of modes
This patch adds chain mode to the list of available modes in coccicheck. Signed-off-by: Sumera Priyadarsini --- Changes in v2: - Change coccinelle to coccicheck as suggested by Julia Lawall. --- scripts/coccicheck | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..6e37cf36caae 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -99,7 +99,7 @@ fi if [ "$MODE" = "" ] ; then if [ "$ONLINE" = "0" ] ; then echo 'You have not explicitly specified the mode to use. Using default "report" mode.' - echo 'Available modes are the following: patch, report, context, org' + echo 'Available modes are the following: patch, report, context, org, chain' echo 'You can specify the mode with "make coccicheck MODE="' echo 'Note however that some modes are not implemented by some semantic patches.' fi -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
[Cocci] [PATCH] scripts: coccicheck: Add chain mode to list of modes
This patch adds chain mode to the list of available modes in coccinelle. Signed-off-by: Sumera Priyadarsini --- scripts/coccicheck | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index e04d328210ac..6e37cf36caae 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -99,7 +99,7 @@ fi if [ "$MODE" = "" ] ; then if [ "$ONLINE" = "0" ] ; then echo 'You have not explicitly specified the mode to use. Using default "report" mode.' - echo 'Available modes are the following: patch, report, context, org' + echo 'Available modes are the following: patch, report, context, org, chain' echo 'You can specify the mode with "make coccicheck MODE="' echo 'Note however that some modes are not implemented by some semantic patches.' fi -- 2.17.1 ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci