Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 05:44:01PM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, 
> > I'll extend
> > this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where 
> > you can
> > specify extra options that the script doesn't specifically support.
> 
> If it is more convenient, you don't actually have to remove --very-quiet.
> You can just put --quiet before --show-trying or --profile.  --quiet will
> override --very-quiet.


How about just:

if [ "$SPFLAGS"  == *"--profile"* -o "$SPFLAGS"  == "--show-trying" ]; then 
FLAGS="--quiet $SPFLAGS"
else
FLAGS="--very-quiet $SPFLAGS"   
fi 

> > > > > > > > > Originally our use of parmap made output, specia files based 
> > > > > > > > > on pids.  Maybe this
> > > > > > > > > is the default for parmap.  I found this completely unusable. 
> > > > > > > > >  I guess one
> > > > > > > > > could look at the dates to see which file is the most recent 
> > > > > > > > > one, but it
> > > > > > > > > seems tedious.  If you are putting the standard output in 
> > > > > > > > > x.out, then put
> > > > > > > > > the standard error in x.err.
> > > > > > > >
> > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > > > >
> > > > > > > What is ${DIR}? and what is $$?
> > > > > >
> > > > > > When you run scripts/coccicheck we take the absolute directory
> > > > > > of it and then go down one level of directory, so in this case it
> > > > > > would be the base directory of the Linux kernel.
> > > > > >
> > > > > > $$ is the PID of the bash script.
> > > > >
> > > > > OK.  I still don't find PIDs useful, but I guess if we are talking 
> > > > > about
> > > > > the entire output of coccicheck, there is not much else to do.  
> > > > > Normally,
> > > > > I don't want these files accumulating, and just write over the old 
> > > > > ones.
> > > >
> > > > Which is why I would much prefer to instead just redirect in coccicheck
> > > > case stderr to stdout from coccinelle. Is that preferred?
> > >
> > > Then things will be merged in strange ways.
> > >
> > > Why not just let the user decide what to do with these things?
> >
> > Sure, what should be the default?
> 
> I would normally just expect standard output and standard error to appear
> randomly on the screen.  That is, no management effort from the tool at
> all.

But the thing is, stderr is ignored now given that a shell script is used
wrapped over a Makefile so if we want what you describe I think we do
have to by default do 2>&1 on the spatch run command.

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> > >
> > >
> > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > >
> > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > > >
> > > > >
> > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > > >
> > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > >
> > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > > >
> > > > > > > > > > I'll redirect stderr to stdout by default when parmap 
> > > > > > > > > > support is used then.
> > > > > > > > >
> > > > > > > > > Usually I put them in different files.
> > > > > > > >
> > > > > > > > We can do that as well but I would only want to deal with 
> > > > > > > > parmap support
> > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where 
> > > > > > > > PID would
> > > > > > > > be the PID of the shell script?
> > > > > > >
> > > > > > > I don't understand the connection with parmap.
> > > > > >
> > > > > > When parmap support is not available the cocciscript will currently
> > > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > > thread I guess.
> > > > >
> > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > > --very-quiet, so there is standard error.
> > > >
> > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > > determine if we want to append or add one per PID... I rather leave that
> > > > stuff as-is and encourage folks to upgrade coccinelle.
> > >
> > > If coccicheck is using --very-quiet, there will not be much stderr of
> > > interest when using parmap either.
> >
> > OK I don't follow. Does coccinelle only direct error to stderr when 
> > --very-quiet
> > is used ? Or does using --very-quiet suppress stderr ?
> 
> --very-quiet suppresses most administrative messages that go to stderr.
> There are still actual errors.  Bu you don't see eg what file is being
> currently processed.

OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll 
extend
this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you 
can
specify extra options that the script doesn't specifically support.

> > > > > > > Originally our use of parmap made output, specia files based on 
> > > > > > > pids.  Maybe this
> > > > > > > is the default for parmap.  I found this completely unusable.  I 
> > > > > > > guess one
> > > > > > > could look at the dates to see which file is the most recent one, 
> > > > > > > but it
> > > > > > > seems tedious.  If you are putting the standard output in x.out, 
> > > > > > > then put
> > > > > > > the standard error in x.err.
> > > > > >
> > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > >
> > > > > What is ${DIR}? and what is $$?
> > > >
> > > > When you run scripts/coccicheck we take the absolute directory
> > > > of it and then go down one level of directory, so in this case it
> > > > would be the base directory of the Linux kernel.
> > > >
> > > > $$ is the PID of the bash script.
> > >
> > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > I don't want these files accumulating, and just write over the old ones.
> >
> > Which is why I would much prefer to instead just redirect in coccicheck
> > case stderr to stdout from coccinelle. Is that preferred?
> 
> Then things will be merged in strange ways.
> 
> Why not just let the user decide what to do with these things?

Sure, what should be the default?

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> >
> >
> > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > >
> > > >
> > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > >
> > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > >
> > > > > >
> > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > >
> > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > >
> > > > > > > > > I'll redirect stderr to stdout by default when parmap support 
> > > > > > > > > is used then.
> > > > > > > >
> > > > > > > > Usually I put them in different files.
> > > > > > >
> > > > > > > We can do that as well but I would only want to deal with parmap 
> > > > > > > support
> > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID 
> > > > > > > would
> > > > > > > be the PID of the shell script?
> > > > > >
> > > > > > I don't understand the connection with parmap.
> > > > >
> > > > > When parmap support is not available the cocciscript will currently
> > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > thread I guess.
> > > >
> > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > --very-quiet, so there is standard error.
> > >
> > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > determine if we want to append or add one per PID... I rather leave that
> > > stuff as-is and encourage folks to upgrade coccinelle.
> >
> > If coccicheck is using --very-quiet, there will not be much stderr of
> > interest when using parmap either.
>
> OK I don't follow. Does coccinelle only direct error to stderr when 
> --very-quiet
> is used ? Or does using --very-quiet suppress stderr ?

--very-quiet suppresses most administrative messages that go to stderr.
There are still actual errors.  Bu you don't see eg what file is being
currently processed.

> > > > > > Originally our use of parmap made output, specia files based on 
> > > > > > pids.  Maybe this
> > > > > > is the default for parmap.  I found this completely unusable.  I 
> > > > > > guess one
> > > > > > could look at the dates to see which file is the most recent one, 
> > > > > > but it
> > > > > > seems tedious.  If you are putting the standard output in x.out, 
> > > > > > then put
> > > > > > the standard error in x.err.
> > > > >
> > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > >
> > > > What is ${DIR}? and what is $$?
> > >
> > > When you run scripts/coccicheck we take the absolute directory
> > > of it and then go down one level of directory, so in this case it
> > > would be the base directory of the Linux kernel.
> > >
> > > $$ is the PID of the bash script.
> >
> > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > the entire output of coccicheck, there is not much else to do.  Normally,
> > I don't want these files accumulating, and just write over the old ones.
>
> Which is why I would much prefer to instead just redirect in coccicheck
> case stderr to stdout from coccinelle. Is that preferred?

Then things will be merged in strange ways.

Why not just let the user decide what to do with these things?

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > 
> > > > > > > > I'll redirect stderr to stdout by default when parmap support 
> > > > > > > > is used then.
> > > > > > > 
> > > > > > > Usually I put them in different files.
> > > > > > 
> > > > > > We can do that as well but I would only want to deal with parmap 
> > > > > > support 
> > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID 
> > > > > > would 
> > > > > > be the PID of the shell script?
> > > > > 
> > > > > I don't understand the connection with parmap.
> > > > 
> > > > When parmap support is not available the cocciscript will currently
> > > > disregard stderr, output is provided as it comes to stdout from each
> > > > thread I guess.
> > > 
> > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> > > --very-quiet, so there is standard error.
> > 
> > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > determine if we want to append or add one per PID... I rather leave that
> > stuff as-is and encourage folks to upgrade coccinelle.
> 
> If coccicheck is using --very-quiet, there will not be much stderr of 
> interest when using parmap either.

OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
is used ? Or does using --very-quiet suppress stderr ?

> I'm not sure to understan the issue about appending.  Appending what to 
> what?
> While Coccinelle is running, you have a directory with the same 
> name as your semantic patch (or a directory name that you specify with 
> --tmp-dir) that has separate files for each core's standard output and 
> standard error.  At the end, when all the code has been treated, 
> Coccinelle writes the successive stdouts to standard output, and the 
> sucessive stderrs to standard error.

Ah I see. OK yeah never mind about appending.

> > > > > Originally our use of parmap made output, specia files based on pids. 
> > > > >  Maybe this 
> > > > > is the default for parmap.  I found this completely unusable.  I 
> > > > > guess one 
> > > > > could look at the dates to see which file is the most recent one, but 
> > > > > it 
> > > > > seems tedious.  If you are putting the standard output in x.out, then 
> > > > > put 
> > > > > the standard error in x.err.
> > > > 
> > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > 
> > > What is ${DIR}? and what is $$?
> > 
> > When you run scripts/coccicheck we take the absolute directory
> > of it and then go down one level of directory, so in this case it
> > would be the base directory of the Linux kernel.
> > 
> > $$ is the PID of the bash script.
> 
> OK.  I still don't find PIDs useful, but I guess if we are talking about 
> the entire output of coccicheck, there is not much else to do.  Normally, 
> I don't want these files accumulating, and just write over the old ones.

Which is why I would much prefer to instead just redirect in coccicheck
case stderr to stdout from coccinelle. Is that preferred?

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > 
> > > > > > 
> > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > 
> > > > > > > I'll redirect stderr to stdout by default when parmap support is 
> > > > > > > used then.
> > > > > > 
> > > > > > Usually I put them in different files.
> > > > > 
> > > > > We can do that as well but I would only want to deal with parmap 
> > > > > support 
> > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID 
> > > > > would 
> > > > > be the PID of the shell script?
> > > > 
> > > > I don't understand the connection with parmap.
> > > 
> > > When parmap support is not available the cocciscript will currently
> > > disregard stderr, output is provided as it comes to stdout from each
> > > thread I guess.
> > 
> > Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> > --very-quiet, so there is standard error.
> 
> OK I'm disegarding the redirect for non-parmap for now but we'd have to
> determine if we want to append or add one per PID... I rather leave that
> stuff as-is and encourage folks to upgrade coccinelle.

If coccicheck is using --very-quiet, there will not be much stderr of 
interest when using parmap either.

I'm not sure to understan the issue about appending.  Appending what to 
what?  While Coccinelle is running, you have a directory with the same 
name as your semantic patch (or a directory name that you specify with 
--tmp-dir) that has separate files for each core's standard output and 
standard error.  At the end, when all the code has been treated, 
Coccinelle writes the successive stdouts to standard output, and the 
sucessive stderrs to standard error.

> > > > Originally our use of parmap made output files based on pids.  Maybe 
> > > > this 
> > > > is the default for parmap.  I found this completely unusable.  I guess 
> > > > one 
> > > > could look at the dates to see which file is the most recent one, but 
> > > > it 
> > > > seems tedious.  If you are putting the standard output in x.out, then 
> > > > put 
> > > > the standard error in x.err.
> > > 
> > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > 
> > What is ${DIR}? and what is $$?
> 
> When you run scripts/coccicheck we take the absolute directory
> of it and then go down one level of directory, so in this case it
> would be the base directory of the Linux kernel.
> 
> $$ is the PID of the bash script.

OK.  I still don't find PIDs useful, but I guess if we are talking about 
the entire output of coccicheck, there is not much else to do.  Normally, 
I don't want these files accumulating, and just write over the old ones.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > I'll redirect stderr to stdout by default when parmap support is used 
> > > > > then.
> > > > 
> > > > Usually I put them in different files.
> > > 
> > > We can do that as well but I would only want to deal with parmap support 
> > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > be the PID of the shell script?
> > 
> > I don't understand the connection with parmap.
> 
> When parmap support is not available the cocciscript will currently
> disregard stderr, output is provided as it comes to stdout from each
> thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
--very-quiet, so there is standard error.

> > Originally our use of parmap made output files based on pids.  Maybe this 
> > is the default for parmap.  I found this completely unusable.  I guess one 
> > could look at the dates to see which file is the most recent one, but it 
> > seems tedious.  If you are putting the standard output in x.out, then put 
> > the standard error in x.err.
> 
> I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > I'll redirect stderr to stdout by default when parmap support is used 
> > > > then.
> > > 
> > > Usually I put them in different files.
> > 
> > We can do that as well but I would only want to deal with parmap support 
> > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > be the PID of the shell script?
> 
> I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently
disregard stderr, output is provided as it comes to stdout from each
thread I guess.

> Originally our use of parmap made output files based on pids.  Maybe this 
> is the default for parmap.  I found this completely unusable.  I guess one 
> could look at the dates to see which file is the most recent one, but it 
> seems tedious.  If you are putting the standard output in x.out, then put 
> the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > > > 
> > > > > > 
> > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > 
> > > > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > > > in practice though this seems to not perform better than
> > > > > > > regular grep however its expected to help with some use cases
> > > > > > > so we use that if you have no other indexing options in place
> > > > > > > available.
> > > > > > > 
> > > > > > > Since git has its own index, support for using 'git grep' has been
> > > > > > > added to Coccinelle, that should on average perform better than
> > > > > > > using the internal cocci grep, and regular grep. Lastly, 
> > > > > > > Coccinelle
> > > > > > > has had support for glimpseindex for a long while, however the
> > > > > > > tool was previously closed source, its now open sourced, and
> > > > > > > provides the best performance, so support that if we can detect
> > > > > > > you have a glimpse index.
> > > > > > > 
> > > > > > > These tests have been run on an 8 core system:
> > > > > > > 
> > > > > > > Before:
> > > > > > > 
> > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > > > $ time make coccicheck MODE=report
> > > > > > > 
> > > > > > > Before this patch with no indexing or anything:
> > > > > > > 
> > > > > > > real16m22.435s
> > > > > > > user128m30.060s
> > > > > > > sys 0m2.712s
> > > > > > > 
> > > > > > > Using coccigrep (after this patch if you have no .git):
> > > > > > > 
> > > > > > > real16m27.650s
> > > > > > > user128m47.904s
> > > > > > > sys 0m2.176s
> > > > > > > 
> > > > > > > If you have .git and therefore use gitgrep:
> > > > > > > 
> > > > > > > real16m21.220s
> > > > > > > user129m30.940s
> > > > > > > sys 0m2.060s
> > > > > > > 
> > > > > > > And if you have a .glimpse_index:
> > > > > > > 
> > > > > > > real16m14.794s
> > > > > > > user128m42.356s
> > > > > > > sys 0m1.880s
> > > > > > 
> > > > > > I don't see any convincing differences in these times.
> > > > > > 
> > > > > > I believe that Coccinelle's internal grep is always used, even with 
> > > > > > no 
> > > > > > option.
> > > > > 
> > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > > > there to use --use-coccigrep if no other index is specified ?
> > > > 
> > > > It has been the default for a long time.
> > > > 
> > > > > > I'm puzzled why glimpse gives no benefit.
> > > > > 
> > > > > Well, slightly better.
> > > > 
> > > > No, it should be much better.  You would have to look at the standard 
> > > > error to see if you are getting any benefit.  There should be very few 
> > > > occurrences of Skipping if you are really using glimpse.  In any case, 
> > > > if 
> > > > you asked for glimpse and it was not able to provide it, there should 
> > > > be 
> > > > warning messages at the top of stderr.
> > > 
> > > I'll redirect stderr to stdout by default when parmap support is used 
> > > then.
> > 
> > Usually I put them in different files.
> 
> We can do that as well but I would only want to deal with parmap support 
> case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> be the PID of the shell script?

I don't understand the connection with parmap.

Originally our use of parmap made output files based on pids.  Maybe this 
is the default for parmap.  I found this completely unusable.  I guess one 
could look at the dates to see which file is the most recent one, but it 
seems tedious.  If you are putting the standard output in x.out, then put 
the standard error in x.err.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Wolfram Sang wrote:
> > 
> > > 
> > > > Is there another scripts/coccinelle/ file I can use to test against to 
> > > > demo
> > > > against glimpse/idutils/gitgrep best?
> > > 
> > > I'd think this one may be a candidate:
> > > 
> > > scripts/coccinelle/misc/irqf_oneshot.cocci
> > > 
> > > Not too many, but quite some matches over the tree.
> > 
> > Seems like a reasonable choice.
> 
> With this one on a 32-core system, I get:
> 
> glimpse:
> real0m6.549s
> user0m49.136s
> sys 0m3.076s
> 
> idutils:
> real0m6.749s
> user0m51.936s
> sys 0m3.876s
> 
> gitgrep:
> real0m6.805s
> user0m51.572s
> sys 0m4.432s
> 
> coccigrep:
> real0m16.369s
> user0m58.712s
> sys 0m5.064s
> 
> I redirected stderr to stdout, and verified glimpse output has:
> 
> glimpse request = request_threaded_irq
> 
> Does this match expectations?

Yes.  I'm not sure that gitgrep would work as well when there are multiple 
keywords.  It may descend to coccigrep.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > > in practice though this seems to not perform better than
> > > > > > regular grep however its expected to help with some use cases
> > > > > > so we use that if you have no other indexing options in place
> > > > > > available.
> > > > > > 
> > > > > > Since git has its own index, support for using 'git grep' has been
> > > > > > added to Coccinelle, that should on average perform better than
> > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > > has had support for glimpseindex for a long while, however the
> > > > > > tool was previously closed source, its now open sourced, and
> > > > > > provides the best performance, so support that if we can detect
> > > > > > you have a glimpse index.
> > > > > > 
> > > > > > These tests have been run on an 8 core system:
> > > > > > 
> > > > > > Before:
> > > > > > 
> > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > > $ time make coccicheck MODE=report
> > > > > > 
> > > > > > Before this patch with no indexing or anything:
> > > > > > 
> > > > > > real16m22.435s
> > > > > > user128m30.060s
> > > > > > sys 0m2.712s
> > > > > > 
> > > > > > Using coccigrep (after this patch if you have no .git):
> > > > > > 
> > > > > > real16m27.650s
> > > > > > user128m47.904s
> > > > > > sys 0m2.176s
> > > > > > 
> > > > > > If you have .git and therefore use gitgrep:
> > > > > > 
> > > > > > real16m21.220s
> > > > > > user129m30.940s
> > > > > > sys 0m2.060s
> > > > > > 
> > > > > > And if you have a .glimpse_index:
> > > > > > 
> > > > > > real16m14.794s
> > > > > > user128m42.356s
> > > > > > sys 0m1.880s
> > > > > 
> > > > > I don't see any convincing differences in these times.
> > > > > 
> > > > > I believe that Coccinelle's internal grep is always used, even with 
> > > > > no 
> > > > > option.
> > > > 
> > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > > there to use --use-coccigrep if no other index is specified ?
> > > 
> > > It has been the default for a long time.
> > > 
> > > > > I'm puzzled why glimpse gives no benefit.
> > > > 
> > > > Well, slightly better.
> > > 
> > > No, it should be much better.  You would have to look at the standard 
> > > error to see if you are getting any benefit.  There should be very few 
> > > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > > you asked for glimpse and it was not able to provide it, there should be 
> > > warning messages at the top of stderr.
> > 
> > I'll redirect stderr to stdout by default when parmap support is used then.
> 
> Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case.
Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of
the shell script?

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Wolfram Sang wrote:
> 
> > 
> > > Is there another scripts/coccinelle/ file I can use to test against to 
> > > demo
> > > against glimpse/idutils/gitgrep best?
> > 
> > I'd think this one may be a candidate:
> > 
> > scripts/coccinelle/misc/irqf_oneshot.cocci
> > 
> > Not too many, but quite some matches over the tree.
> 
> Seems like a reasonable choice.

With this one on a 32-core system, I get:

glimpse:
real0m6.549s
user0m49.136s
sys 0m3.076s

idutils:
real0m6.749s
user0m51.936s
sys 0m3.876s

gitgrep:
real0m6.805s
user0m51.572s
sys 0m4.432s

coccigrep:
real0m16.369s
user0m58.712s
sys 0m5.064s

I redirected stderr to stdout, and verified glimpse output has:

glimpse request = request_threaded_irq

Does this match expectations?

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > in practice though this seems to not perform better than
> > > > > regular grep however its expected to help with some use cases
> > > > > so we use that if you have no other indexing options in place
> > > > > available.
> > > > > 
> > > > > Since git has its own index, support for using 'git grep' has been
> > > > > added to Coccinelle, that should on average perform better than
> > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > has had support for glimpseindex for a long while, however the
> > > > > tool was previously closed source, its now open sourced, and
> > > > > provides the best performance, so support that if we can detect
> > > > > you have a glimpse index.
> > > > > 
> > > > > These tests have been run on an 8 core system:
> > > > > 
> > > > > Before:
> > > > > 
> > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > $ time make coccicheck MODE=report
> > > > > 
> > > > > Before this patch with no indexing or anything:
> > > > > 
> > > > > real16m22.435s
> > > > > user128m30.060s
> > > > > sys 0m2.712s
> > > > > 
> > > > > Using coccigrep (after this patch if you have no .git):
> > > > > 
> > > > > real16m27.650s
> > > > > user128m47.904s
> > > > > sys 0m2.176s
> > > > > 
> > > > > If you have .git and therefore use gitgrep:
> > > > > 
> > > > > real16m21.220s
> > > > > user129m30.940s
> > > > > sys 0m2.060s
> > > > > 
> > > > > And if you have a .glimpse_index:
> > > > > 
> > > > > real16m14.794s
> > > > > user128m42.356s
> > > > > sys 0m1.880s
> > > > 
> > > > I don't see any convincing differences in these times.
> > > > 
> > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > option.
> > > 
> > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > there to use --use-coccigrep if no other index is specified ?
> > 
> > It has been the default for a long time.
> > 
> > > > I'm puzzled why glimpse gives no benefit.
> > > 
> > > Well, slightly better.
> > 
> > No, it should be much better.  You would have to look at the standard 
> > error to see if you are getting any benefit.  There should be very few 
> > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > you asked for glimpse and it was not able to provide it, there should be 
> > warning messages at the top of stderr.
> 
> I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Wolfram Sang wrote:

> 
> > Is there another scripts/coccinelle/ file I can use to test against to demo
> > against glimpse/idutils/gitgrep best?
> 
> I'd think this one may be a candidate:
> 
> scripts/coccinelle/misc/irqf_oneshot.cocci
> 
> Not too many, but quite some matches over the tree.

Seems like a reasonable choice.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > support to make use of its own enhanced "grep" mechanisms
> > > > instead of using regular grep for searching code 'coccigrep',
> > > > in practice though this seems to not perform better than
> > > > regular grep however its expected to help with some use cases
> > > > so we use that if you have no other indexing options in place
> > > > available.
> > > > 
> > > > Since git has its own index, support for using 'git grep' has been
> > > > added to Coccinelle, that should on average perform better than
> > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > has had support for glimpseindex for a long while, however the
> > > > tool was previously closed source, its now open sourced, and
> > > > provides the best performance, so support that if we can detect
> > > > you have a glimpse index.
> > > > 
> > > > These tests have been run on an 8 core system:
> > > > 
> > > > Before:
> > > > 
> > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > $ time make coccicheck MODE=report
> > > > 
> > > > Before this patch with no indexing or anything:
> > > > 
> > > > real16m22.435s
> > > > user128m30.060s
> > > > sys 0m2.712s
> > > > 
> > > > Using coccigrep (after this patch if you have no .git):
> > > > 
> > > > real16m27.650s
> > > > user128m47.904s
> > > > sys 0m2.176s
> > > > 
> > > > If you have .git and therefore use gitgrep:
> > > > 
> > > > real16m21.220s
> > > > user129m30.940s
> > > > sys 0m2.060s
> > > > 
> > > > And if you have a .glimpse_index:
> > > > 
> > > > real16m14.794s
> > > > user128m42.356s
> > > > sys 0m1.880s
> > > 
> > > I don't see any convincing differences in these times.
> > > 
> > > I believe that Coccinelle's internal grep is always used, even with no 
> > > option.
> > 
> > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > there to use --use-coccigrep if no other index is specified ?
> 
> It has been the default for a long time.
> 
> > > I'm puzzled why glimpse gives no benefit.
> > 
> > Well, slightly better.
> 
> No, it should be much better.  You would have to look at the standard 
> error to see if you are getting any benefit.  There should be very few 
> occurrences of Skipping if you are really using glimpse.  In any case, if 
> you asked for glimpse and it was not able to provide it, there should be 
> warning messages at the top of stderr.

I'll redirect stderr to stdout by default when parmap support is used then.

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

> Is there another scripts/coccinelle/ file I can use to test against to demo
> against glimpse/idutils/gitgrep best?

I'd think this one may be a candidate:

scripts/coccinelle/misc/irqf_oneshot.cocci

Not too many, but quite some matches over the tree.



signature.asc
Description: PGP signature
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Sat, Jun 11, 2016 at 07:24:51AM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Wolfram Sang wrote:
> 
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> 
> Coccinelle includes a script scripts/idutils_index.sh
> 
> This does mkid -i C --output .id-utils.index *

I'll add support for detecting both. An issue of course is if any of these
indexes grows stale. So I'll advise against these and recommend gitgrep
unless the user has a hook to update index regularly.

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Sat, Jun 11, 2016 at 07:54:39AM +0200, Julia Lawall wrote:
> 
> 
> On Sat, 11 Jun 2016, Wolfram Sang wrote:
> 
> > 
> > > real16m11.692s
> > > user127m50.388s
> > > sys 0m2.168s
> > 
> > That's better but not a magnitude, I wonder.
> 
> I think that it is because the filtering that Coccinelle does already 
> works pretty well, and there are quite a lot of files (7514) that contain 
> kfree.

Is there another scripts/coccinelle/ file I can use to test against to demo
against glimpse/idutils/gitgrep best?

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

> works pretty well, and there are quite a lot of files (7514) that contain 
> kfree.

Ah, kfree. That explains, I missed that info.



signature.asc
Description: PGP signature
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Sat, 11 Jun 2016, Wolfram Sang wrote:

> 
> > It's not as efficient as glimpse because the query language is simpler.  
> 
> Interesting, what is missing compared to glimpse?

Glimpse allows queries that are arbitrary formulas, up to a limited level 
of complexity, involving both && and ||.  For idutils, Coccinelle runs lid 
on each of the tokens in the formula, and then does unions and 
intersections on the result.

julia

> > So more filtering has to be done at the ocaml level.  But it's probably 
> > fine in most cases.
> 
> For me, it has two advantages over glimpse:
> 
> a) it is in the debian package repository
> b) the same database can be used with the code browser 'seascope'
>which can do nice things by feeding ctags on the fly with data
>from id-utils.
> 
> Mileages vary, of course, just wanted to mention it to give pointers.
> 
> 
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

> It's not as efficient as glimpse because the query language is simpler.  

Interesting, what is missing compared to glimpse?

> So more filtering has to be done at the ocaml level.  But it's probably 
> fine in most cases.

For me, it has two advantages over glimpse:

a) it is in the debian package repository
b) the same database can be used with the code browser 'seascope'
   which can do nice things by feeding ctags on the fly with data
   from id-utils.

Mileages vary, of course, just wanted to mention it to give pointers.



signature.asc
Description: PGP signature
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[SF Markus Elfring]

> in practice though this seems to not perform better than
> regular grep however its expected to help with some use cases
> so we use that if you have no other indexing options in place
> available.

Would you like to fix a typo in this paragraph?

Regards,
Markus
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Sat, 11 Jun 2016, Wolfram Sang wrote:

> 
> > real16m11.692s
> > user127m50.388s
> > sys 0m2.168s
> 
> That's better but not a magnitude, I wonder.

I think that it is because the filtering that Coccinelle does already 
works pretty well, and there are quite a lot of files (7514) that contain 
kfree.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

> real16m11.692s
> user127m50.388s
> sys 0m2.168s

That's better but not a magnitude, I wonder.



signature.asc
Description: PGP signature
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Sat, 11 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> > 
> 
> Sweet, testing that now.

It's not as efficient as glimpse because the query language is simpler.  
So more filtering has to be done at the ocaml level.  But it's probably 
fine in most cases.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote:
> > > > Well, slightly better.
> > > 
> > > No, it should be much better.  You would have to look at the standard 
> > 
> > I use id-utils regularly and it is indeed at least a magnitude better.
> > The indexing often pays off already with the first coccinelle run for
> > me. Highly recommended.
> 
> AFAICT coccinelle does not have integration support for id-utils though.

Yes it does, for years.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Sat, Jun 11, 2016 at 12:08:32AM +0200, Luis R. Rodriguez wrote:
> On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> > 
> 
> Sweet, testing that now.

Boooyah :D

real16m11.692s
user127m50.388s
sys 0m2.168s

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

> AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.



signature.asc
Description: PGP signature
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote:
> > > Well, slightly better.
> > 
> > No, it should be much better.  You would have to look at the standard 
> 
> I use id-utils regularly and it is indeed at least a magnitude better.
> The indexing often pays off already with the first coccinelle run for
> me. Highly recommended.

AFAICT coccinelle does not have integration support for id-utils though.

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

> > Well, slightly better.
> 
> No, it should be much better.  You would have to look at the standard 

I use id-utils regularly and it is indeed at least a magnitude better.
The indexing often pays off already with the first coccinelle run for
me. Highly recommended.



signature.asc
Description: PGP signature
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > Enable indexing optimizations heuristics. Coccinelle has
> > > support to make use of its own enhanced "grep" mechanisms
> > > instead of using regular grep for searching code 'coccigrep',
> > > in practice though this seems to not perform better than
> > > regular grep however its expected to help with some use cases
> > > so we use that if you have no other indexing options in place
> > > available.
> > > 
> > > Since git has its own index, support for using 'git grep' has been
> > > added to Coccinelle, that should on average perform better than
> > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > has had support for glimpseindex for a long while, however the
> > > tool was previously closed source, its now open sourced, and
> > > provides the best performance, so support that if we can detect
> > > you have a glimpse index.
> > > 
> > > These tests have been run on an 8 core system:
> > > 
> > > Before:
> > > 
> > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > $ time make coccicheck MODE=report
> > > 
> > > Before this patch with no indexing or anything:
> > > 
> > > real16m22.435s
> > > user128m30.060s
> > > sys 0m2.712s
> > > 
> > > Using coccigrep (after this patch if you have no .git):
> > > 
> > > real16m27.650s
> > > user128m47.904s
> > > sys 0m2.176s
> > > 
> > > If you have .git and therefore use gitgrep:
> > > 
> > > real16m21.220s
> > > user129m30.940s
> > > sys 0m2.060s
> > > 
> > > And if you have a .glimpse_index:
> > > 
> > > real16m14.794s
> > > user128m42.356s
> > > sys 0m1.880s
> > 
> > I don't see any convincing differences in these times.
> > 
> > I believe that Coccinelle's internal grep is always used, even with no 
> > option.
> 
> Ah that would explain it. This uses coccinelle 1.0.5, is the default
> there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

> > I'm puzzled why glimpse gives no benefit.
> 
> Well, slightly better.

No, it should be much better.  You would have to look at the standard 
error to see if you are getting any benefit.  There should be very few 
occurrences of Skipping if you are really using glimpse.  In any case, if 
you asked for glimpse and it was not able to provide it, there should be 
warning messages at the top of stderr.

julia
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> 
> > Enable indexing optimizations heuristics. Coccinelle has
> > support to make use of its own enhanced "grep" mechanisms
> > instead of using regular grep for searching code 'coccigrep',
> > in practice though this seems to not perform better than
> > regular grep however its expected to help with some use cases
> > so we use that if you have no other indexing options in place
> > available.
> > 
> > Since git has its own index, support for using 'git grep' has been
> > added to Coccinelle, that should on average perform better than
> > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > has had support for glimpseindex for a long while, however the
> > tool was previously closed source, its now open sourced, and
> > provides the best performance, so support that if we can detect
> > you have a glimpse index.
> > 
> > These tests have been run on an 8 core system:
> > 
> > Before:
> > 
> > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > $ time make coccicheck MODE=report
> > 
> > Before this patch with no indexing or anything:
> > 
> > real16m22.435s
> > user128m30.060s
> > sys 0m2.712s
> > 
> > Using coccigrep (after this patch if you have no .git):
> > 
> > real16m27.650s
> > user128m47.904s
> > sys 0m2.176s
> > 
> > If you have .git and therefore use gitgrep:
> > 
> > real16m21.220s
> > user129m30.940s
> > sys 0m2.060s
> > 
> > And if you have a .glimpse_index:
> > 
> > real16m14.794s
> > user128m42.356s
> > sys 0m1.880s
> 
> I don't see any convincing differences in these times.
> 
> I believe that Coccinelle's internal grep is always used, even with no 
> option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default
there to use --use-coccigrep if no other index is specified ?

> I'm puzzled why glimpse gives no benefit.

Well, slightly better.

  Luis
___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci

[Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

Enable indexing optimizations heuristics. Coccinelle has
support to make use of its own enhanced "grep" mechanisms
instead of using regular grep for searching code 'coccigrep',
in practice though this seems to not perform better than
regular grep however its expected to help with some use cases
so we use that if you have no other indexing options in place
available.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal cocci grep, and regular grep. Lastly, Coccinelle
has had support for glimpseindex for a long while, however the
tool was previously closed source, its now open sourced, and
provides the best performance, so support that if we can detect
you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real16m22.435s
user128m30.060s
sys 0m2.712s

Using coccigrep (after this patch if you have no .git):

real16m27.650s
user128m47.904s
sys 0m2.176s

If you have .git and therefore use gitgrep:

real16m21.220s
user129m30.940s
sys 0m2.060s

And if you have a .glimpse_index:

real16m14.794s
user128m42.356s
sys 0m1.880s

Signed-off-by: Luis R. Rodriguez 
---
 scripts/coccicheck | 24 
 1 file changed, 24 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index eeb5fdc142ca..f31c9a152559 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +17,20 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# 0. --use-glimpse currently outperforms all. Refer
+#to scripts/glimpse.sh for details.
+# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
+# 2. Use --use-coccigrep if no indexing options are available and your
+#version of coccinelle supports it
+USE_GLIMPSE="no"
+$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f 
$DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+USE_GITGREP="no"
+$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && 
USE_GITGREP="yes"
+
+USE_COCCIGREP="no"
+$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +105,14 @@ else
OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+if [ "$USE_GLIMPSE" = "yes" ]; then
+   OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_GITGREP" = "yes" ]; then
+   OPTIONS="$OPTIONS --use-gitgrep"
+elif [ "$USE_COCCIGREP" = "yes" ]; then
+   OPTIONS="$OPTIONS --use-coccigrep"
+fi
+
 run_cmd_paramap() {
if [ $VERBOSE -ne 0 ] ; then
echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

___
Cocci mailing list
Cocci@systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci