Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > > > > support to make use of its own enhanced "grep" mechanisms > > > > > > > instead of using regular grep for searching code 'coccigrep', > > > > > > > in practice though this seems to not perform better than > > > > > > > regular grep however its expected to help with some use cases > > > > > > > so we use that if you have no other indexing options in place > > > > > > > available. > > > > > > > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > > > > added to Coccinelle, that should on average perform better than > > > > > > > using the internal cocci grep, and regular grep. Lastly, > > > > > > > Coccinelle > > > > > > > has had support for glimpseindex for a long while, however the > > > > > > > tool was previously closed source, its now open sourced, and > > > > > > > provides the best performance, so support that if we can detect > > > > > > > you have a glimpse index. > > > > > > > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > > > > > > > Before: > > > > > > > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > > > > $ time make coccicheck MODE=report > > > > > > > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > > > > > > > real16m22.435s > > > > > > > user128m30.060s > > > > > > > sys 0m2.712s > > > > > > > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > > > > > > > real16m27.650s > > > > > > > user128m47.904s > > > > > > > sys 0m2.176s > > > > > > > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > > > > > > > real16m21.220s > > > > > > > user129m30.940s > > > > > > > sys 0m2.060s > > > > > > > > > > > > > > And if you have a .glimpse_index: > > > > > > > > > > > > > > real16m14.794s > > > > > > > user128m42.356s > > > > > > > sys 0m1.880s > > > > > > > > > > > > I don't see any convincing differences in these times. > > > > > > > > > > > > I believe that Coccinelle's internal grep is always used, even with > > > > > > no > > > > > > option. > > > > > > > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > > > > there to use --use-coccigrep if no other index is specified ? > > > > > > > > It has been the default for a long time. > > > > > > > > > > I'm puzzled why glimpse gives no benefit. > > > > > > > > > > Well, slightly better. > > > > > > > > No, it should be much better. You would have to look at the standard > > > > error to see if you are getting any benefit. There should be very few > > > > occurrences of Skipping if you are really using glimpse. In any case, > > > > if > > > > you asked for glimpse and it was not able to provide it, there should > > > > be > > > > warning messages at the top of stderr. > > > > > > I'll redirect stderr to stdout by default when parmap support is used > > > then. > > > > Usually I put them in different files. > > We can do that as well but I would only want to deal with parmap support > case. Any preference? How about .coccicheck.stderr.$PID where PID would > be the PID of the shell script? I don't understand the connection with parmap. Originally our use of parmap made output files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err. julia ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote: > > > > > > On Mon, 13 Jun 2016, Wolfram Sang wrote: > > > > > > > > > Is there another scripts/coccinelle/ file I can use to test against to > > > > demo > > > > against glimpse/idutils/gitgrep best? > > > > > > I'd think this one may be a candidate: > > > > > > scripts/coccinelle/misc/irqf_oneshot.cocci > > > > > > Not too many, but quite some matches over the tree. > > > > Seems like a reasonable choice. > > With this one on a 32-core system, I get: > > glimpse: > real0m6.549s > user0m49.136s > sys 0m3.076s > > idutils: > real0m6.749s > user0m51.936s > sys 0m3.876s > > gitgrep: > real0m6.805s > user0m51.572s > sys 0m4.432s > > coccigrep: > real0m16.369s > user0m58.712s > sys 0m5.064s > > I redirected stderr to stdout, and verified glimpse output has: > > glimpse request = request_threaded_irq > > Does this match expectations? Yes. I'm not sure that gitgrep would work as well when there are multiple keywords. It may descend to coccigrep. julia ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > > > support to make use of its own enhanced "grep" mechanisms > > > > > > instead of using regular grep for searching code 'coccigrep', > > > > > > in practice though this seems to not perform better than > > > > > > regular grep however its expected to help with some use cases > > > > > > so we use that if you have no other indexing options in place > > > > > > available. > > > > > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > > > added to Coccinelle, that should on average perform better than > > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > > > has had support for glimpseindex for a long while, however the > > > > > > tool was previously closed source, its now open sourced, and > > > > > > provides the best performance, so support that if we can detect > > > > > > you have a glimpse index. > > > > > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > > > > > Before: > > > > > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > > > $ time make coccicheck MODE=report > > > > > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > > > > > real16m22.435s > > > > > > user128m30.060s > > > > > > sys 0m2.712s > > > > > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > > > > > real16m27.650s > > > > > > user128m47.904s > > > > > > sys 0m2.176s > > > > > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > > > > > real16m21.220s > > > > > > user129m30.940s > > > > > > sys 0m2.060s > > > > > > > > > > > > And if you have a .glimpse_index: > > > > > > > > > > > > real16m14.794s > > > > > > user128m42.356s > > > > > > sys 0m1.880s > > > > > > > > > > I don't see any convincing differences in these times. > > > > > > > > > > I believe that Coccinelle's internal grep is always used, even with > > > > > no > > > > > option. > > > > > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > > > there to use --use-coccigrep if no other index is specified ? > > > > > > It has been the default for a long time. > > > > > > > > I'm puzzled why glimpse gives no benefit. > > > > > > > > Well, slightly better. > > > > > > No, it should be much better. You would have to look at the standard > > > error to see if you are getting any benefit. There should be very few > > > occurrences of Skipping if you are really using glimpse. In any case, if > > > you asked for glimpse and it was not able to provide it, there should be > > > warning messages at the top of stderr. > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > Usually I put them in different files. We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script? Luis ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote: > > > On Mon, 13 Jun 2016, Wolfram Sang wrote: > > > > > > Is there another scripts/coccinelle/ file I can use to test against to > > > demo > > > against glimpse/idutils/gitgrep best? > > > > I'd think this one may be a candidate: > > > > scripts/coccinelle/misc/irqf_oneshot.cocci > > > > Not too many, but quite some matches over the tree. > > Seems like a reasonable choice. With this one on a 32-core system, I get: glimpse: real0m6.549s user0m49.136s sys 0m3.076s idutils: real0m6.749s user0m51.936s sys 0m3.876s gitgrep: real0m6.805s user0m51.572s sys 0m4.432s coccigrep: real0m16.369s user0m58.712s sys 0m5.064s I redirected stderr to stdout, and verified glimpse output has: glimpse request = request_threaded_irq Does this match expectations? Luis ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > > support to make use of its own enhanced "grep" mechanisms > > > > > instead of using regular grep for searching code 'coccigrep', > > > > > in practice though this seems to not perform better than > > > > > regular grep however its expected to help with some use cases > > > > > so we use that if you have no other indexing options in place > > > > > available. > > > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > > added to Coccinelle, that should on average perform better than > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > > has had support for glimpseindex for a long while, however the > > > > > tool was previously closed source, its now open sourced, and > > > > > provides the best performance, so support that if we can detect > > > > > you have a glimpse index. > > > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > > > Before: > > > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > > $ time make coccicheck MODE=report > > > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > > > real16m22.435s > > > > > user128m30.060s > > > > > sys 0m2.712s > > > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > > > real16m27.650s > > > > > user128m47.904s > > > > > sys 0m2.176s > > > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > > > real16m21.220s > > > > > user129m30.940s > > > > > sys 0m2.060s > > > > > > > > > > And if you have a .glimpse_index: > > > > > > > > > > real16m14.794s > > > > > user128m42.356s > > > > > sys 0m1.880s > > > > > > > > I don't see any convincing differences in these times. > > > > > > > > I believe that Coccinelle's internal grep is always used, even with no > > > > option. > > > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > > there to use --use-coccigrep if no other index is specified ? > > > > It has been the default for a long time. > > > > > > I'm puzzled why glimpse gives no benefit. > > > > > > Well, slightly better. > > > > No, it should be much better. You would have to look at the standard > > error to see if you are getting any benefit. There should be very few > > occurrences of Skipping if you are really using glimpse. In any case, if > > you asked for glimpse and it was not able to provide it, there should be > > warning messages at the top of stderr. > > I'll redirect stderr to stdout by default when parmap support is used then. Usually I put them in different files. julia ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Mon, 13 Jun 2016, Wolfram Sang wrote: > > > Is there another scripts/coccinelle/ file I can use to test against to demo > > against glimpse/idutils/gitgrep best? > > I'd think this one may be a candidate: > > scripts/coccinelle/misc/irqf_oneshot.cocci > > Not too many, but quite some matches over the tree. Seems like a reasonable choice. julia ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > support to make use of its own enhanced "grep" mechanisms > > > > instead of using regular grep for searching code 'coccigrep', > > > > in practice though this seems to not perform better than > > > > regular grep however its expected to help with some use cases > > > > so we use that if you have no other indexing options in place > > > > available. > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > added to Coccinelle, that should on average perform better than > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > has had support for glimpseindex for a long while, however the > > > > tool was previously closed source, its now open sourced, and > > > > provides the best performance, so support that if we can detect > > > > you have a glimpse index. > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > Before: > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > $ time make coccicheck MODE=report > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > real16m22.435s > > > > user128m30.060s > > > > sys 0m2.712s > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > real16m27.650s > > > > user128m47.904s > > > > sys 0m2.176s > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > real16m21.220s > > > > user129m30.940s > > > > sys 0m2.060s > > > > > > > > And if you have a .glimpse_index: > > > > > > > > real16m14.794s > > > > user128m42.356s > > > > sys 0m1.880s > > > > > > I don't see any convincing differences in these times. > > > > > > I believe that Coccinelle's internal grep is always used, even with no > > > option. > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > there to use --use-coccigrep if no other index is specified ? > > It has been the default for a long time. > > > > I'm puzzled why glimpse gives no benefit. > > > > Well, slightly better. > > No, it should be much better. You would have to look at the standard > error to see if you are getting any benefit. There should be very few > occurrences of Skipping if you are really using glimpse. In any case, if > you asked for glimpse and it was not able to provide it, there should be > warning messages at the top of stderr. I'll redirect stderr to stdout by default when parmap support is used then. Luis ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
> Is there another scripts/coccinelle/ file I can use to test against to demo > against glimpse/idutils/gitgrep best? I'd think this one may be a candidate: scripts/coccinelle/misc/irqf_oneshot.cocci Not too many, but quite some matches over the tree. signature.asc Description: PGP signature ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Sat, Jun 11, 2016 at 07:24:51AM +0200, Julia Lawall wrote: > > > On Fri, 10 Jun 2016, Wolfram Sang wrote: > > > > AFAICT coccinelle does not have integration support for id-utils though. > > > > I used it just today ;) -- "--use-idutils ./ID" > > > > ID was generated with simple 'mkid -s'. > > Coccinelle includes a script scripts/idutils_index.sh > > This does mkid -i C --output .id-utils.index * I'll add support for detecting both. An issue of course is if any of these indexes grows stale. So I'll advise against these and recommend gitgrep unless the user has a hook to update index regularly. Luis ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options
On Sat, Jun 11, 2016 at 07:54:39AM +0200, Julia Lawall wrote: > > > On Sat, 11 Jun 2016, Wolfram Sang wrote: > > > > > > real16m11.692s > > > user127m50.388s > > > sys 0m2.168s > > > > That's better but not a magnitude, I wonder. > > I think that it is because the filtering that Coccinelle does already > works pretty well, and there are quite a lot of files (7514) that contain > kfree. Is there another scripts/coccinelle/ file I can use to test against to demo against glimpse/idutils/gitgrep best? Luis ___ Cocci mailing list Cocci@systeme.lip6.fr https://systeme.lip6.fr/mailman/listinfo/cocci
