Re: [PATCH v4 1/2] scripts: Support compiled source, improved precise

2020-05-21 Thread Masahiro Yamada
On Fri, May 15, 2020 at 2:10 AM xujialu  wrote:
>
> Sorry for replying so late.
>
> 
>
> I usually don't run scripts/tags.sh directly. But one day i checked git
> log of scripts/tags.sh, and found this commit c69ef1c87b8c said we may
> run it directly. Then i must took care of that.
>
> Here are some cases that i should write clearly before:
> (I omit COMPILED_SOURCE=1 here just for clear and distinct)
>
> 1) make; make gtags;
> 2) make; ./scripts/tags.sh gtags;
> 3) make O=123; make O=123 gtags;
> 4) make O=123; make gtags;
> 5) make O=123; ./scripts/tags.sh gtags;
> 6) make O=/path/out/of/kernel/; make O=/path/out/of/kernel/ gtags;
> 7) make O=/path/out/of/kernel/; SOMETHING ./scripts/tags.sh gtags;
>
> Assume that we just change directory into kernel root directory and vim
> a source file:
>
> case 1): We have GTAGS generated in current directory, no problem;
> In this case: tree=
> case 2): Same as case 1), no problem;
> In this case: tree=

... if you set SRCARCH.

If SRCARCH is unset, it will get a warning.

$ ./scripts/tags.sh  gtags
find: ‘arch/*.[chS]/’: No such file or directory




> case 3): GTAGS is generated in directory 123; Here comes the problem,
>  gtags will give error "Segmentation fault" (eg. global-5.7.1)
>  or give warnning "is out of source tree." (eg. global-6.6.3-2)
>  because 'make O=123' changed to directory 123 and our cute
>  source files is in ../ and gtags seems do not like this path
>  begin with '../', actually it's not an subdiretories for 123;
>  If above situation is not persuasive, then consider one may
>  want generate gtags.files contains files without '../' in
>  kernel root directory, so that gtags could be useful;
>  And this is why case 4) exist, if case 4) is really bad idea
>  then we must have another way to do this - case 5);
> In this case: tree=../


This is a problem of GNU Global, not of our build system.


"Warning: ... is out of source tree." is listed in
the known bugs:

https://www.gnu.org/software/global/bugs.html


Do not mess up our script.


> case 4): This is not good when we 'make O=123 distclean';
> In this case: tree=../



Of course, "make O=123 distclean" cannot clean up
build artifacts created by "make gtags".

So, what problem are you addressing?



> case 5): Find file '.config' in directory 123, then collect files with
>  path just in current directly; No problem;
> In this case: tree=../

You are misunderstanding.

See the comment at line 8.

# Uses the following environment variables:
# SUBARCH, SRCARCH, srctree


If you want to run this script directly,
you must set all the mentioned environment variables correctly,
and also run this script in the correct working directory.

It will create tag files in the current working directory.
^^

Do not change the working directory internally.





> case 6): What if KBUILD_OUTPUT is out of kernel directory? Assume that
>  we get a gtags.files in that directly, in the gtags.files, the
>  file path all begin with full path, guess what, gtags will give
>  the error or warnning described above. Why don't we just
>  generate GTAGS with relative path in kernel root directory?
> In this case: tree=/path/out/of/kernel/

This is the intended behavior for the other TAGS, tags, cscope.

Again, this is a problem of GNU global.


The general rule is like this:

Tag files should be always output to the separate object tree
if O= is given, then file paths should point to the source
tree with either relative or absolute paths.

This is because the source tree is not always writable.
The source tree might be delivered in a DVD-ROM, read-only mounted nfs,
or located under /usr/src/ which is installed by distro source package.




>
> >
> > +   SRCTREE=$(realpath ${tree}.)
> > +
> > +   cd $(dirname $(find -name .config -print -quit).)
> >
> > Why is this needed?
>
> In case 5), the path of source files collected in .cmd files is some
> begin with '../' and some with full path, eg. /usr/include/stdio.h, we
> must change to directory 123 as 'make O=123 gtags' does so that we could
> use same method (described bellow) in both cases.

No.
scripts/tags.sh must be run in the object tree in this case.

5) make O=123; cd 123; ../scripts/tags.sh srctree=.. SRCARCH=x86 gtags




> > Why is --relative-to=${SRCTREE} needed?
> >
> > You are dropping ${SRCTREE} and adding ${ABSPWD}${tree}.
> > I do not understand what this is doing back-and-forth.
>
> These .cmd files also contain the default include dir (eg.
> /usr/include/stdio.h) and even compiler's header files, try make
> ARCH=arm.


I know. Probably, that would not happen
because the following patch is queued up.
https://patchwork.kernel.org/patch/11505807/




> We should first collect files 

Re: [PATCH v4 1/2] scripts: Support compiled source, improved precise

2020-05-11 Thread Masahiro Yamada
On Sat, May 2, 2020 at 2:27 PM xujialu  wrote:
>
> Original 'COMPILED_SOURCE=1 make cscope' collects nearly 3 files
> include too many unused files, this patch precisely collects source
> files from *.cmd, in this case just 3000 files include dts and dtsi.
>
> Usage:
>   1) COMPILED_SOURCE=1   make {cscope,gtags}
>   2) COMPILED_SOURCE=1 KBUILD_ABS_SRCTREE=1  make {cscope,gtags}
>   3) COMPILED_SOURCE=1  ./scripts/tags.sh {cscope,gtags}
>   4) COMPILED_SOURCE=1 ABSPWD=$PWD/ ./scripts/tags.sh {cscope,gtags}
>
> Signed-off-by: xujialu 
> ---
>  scripts/tags.sh | 36 ++--
>  1 file changed, 22 insertions(+), 14 deletions(-)
>
> diff --git a/scripts/tags.sh b/scripts/tags.sh
> index 4e18ae5282a6..941a5c61d343 100755
> --- a/scripts/tags.sh
> +++ b/scripts/tags.sh
> @@ -89,22 +89,30 @@ all_sources()
> find_other_sources '*.[chS]'
>  }
>
> +# COMPILED_SOURCE=1   make {cscope,gtags}
> +# COMPILED_SOURCE=1 KBUILD_ABS_SRCTREE=1  make {cscope,gtags}
> +# COMPILED_SOURCE=1  ./scripts/tags.sh {cscope,gtags}
> +# COMPILED_SOURCE=1 ABSPWD=$PWD/ ./scripts/tags.sh {cscope,gtags}


These comment are misleading since this sounds like
is is only usef for cscope, gtags.


Please do not introduce a new variable ABSPWD, which is unneeded.
This is a rare use-case, but if you want to run this script directly,
you must set the variables described at line 9 properly.




> +xtags_juggle_list()
> +{
> +   SRCTREE=$(realpath ${tree}.)
> +
> +   cd $(dirname $(find -name .config -print -quit).)


Why is this needed?

You are already in objtree
when this script is being run.

If you handle the objects built with O= option,
you need to do 'make O=... gtags'.

> +
> +   realpath -e --relative-to=${SRCTREE} $(find -name "*.cmd" -exec \
> +   grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ 
> |
> +   awk '!a[$0]++') include/generated/autoconf.h |
> +   sed -e "/\.\./d" -e "s,^,${ABSPWD}${tree},"
> +}

Why is --relative-to=${SRCTREE} needed?

You are dropping ${SRCTREE} and adding ${ABSPWD}${tree}.
I do not understand what this is doing back-and-forth.



Lastly, the file order is currently carefully crafted
but this patch would make it random-ordered.

I am afraid the following commit would be broken.




commit f81b1be40c44b33b9706d64c117edd29e627ad12 (HEAD)
Author: Guennadi Liakhovetski 
Date:   Mon Feb 8 00:25:59 2010 +0100

tags: include headers before source files

Currently looking up a structure definition in TAGS / tags takes one to
one of multiple "static struct X" definitions in arch sources, which makes
it for many structs practically impossible to get to the required header.
This patch changes the order of sources being tagged to first scan
architecture includes, then the top-level include/ directory, and only
then the rest. It also takes into account, that many architectures have
more than one include directory, i.e., not only arch/$ARCH/include, but
also arch/$ARCH/mach-X/include etc.

Signed-off-by: Guennadi Liakhovetski 
Reviewed-by: WANG Cong 
[mma...@suse.cz: fix 'var+=text' bashism]
Signed-off-by: Michal Marek 

> +
>  all_compiled_sources()
>  {
> -   for i in $(all_sources); do
> -   case "$i" in
> -   *.[cS])
> -   j=${i/\.[cS]/\.o}
> -   j="${j#$tree}"
> -   if [ -e $j ]; then
> -   echo $i
> -   fi
> -   ;;
> -   *)
> -   echo $i
> -   ;;
> -   esac
> -   done
> +   # Consider 'git ls-files' features:
> +   #   1) sort and uniq target files
> +   #   2) limit target files by index
> +   # git ls-files $(xtags_juggle_list)


How is this related to this ?






> +
> +   xtags_juggle_list | sort -u
>  }
>
>  all_target_sources()
> --
> 2.20.1
>


--
Best Regards
Masahiro Yamada