Re: [openstack-dev] [bashate] .bashateignore

[Ian Wienand]

On 09/03/2014 11:32 AM, Robert Collins wrote:

if-has-bash-hashbang-and-is-versioned-then-bashate-it?


That misses library files that aren't execed and have no #!

This might be an appropriate rule for test infrastructure to generate a 
list for their particular project, but IMO I don't think we need to 
start building that logic into bashate


-i

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Dean Troyer]

On Tue, Sep 2, 2014 at 8:32 PM, Robert Collins 
wrote:

> Well, git knows all the files in-tree, right? Or am I missing something
> here?
>
> if-has-bash-hashbang-and-is-versioned-then-bashate-it?


It's not quote that simple, none of the include files have a shebang line;
I've always felt that having that in an include file is an indication that
the file is (also) a stand-alone script.   Shocco (the docs processor)
wants one too.

I think I've given up attempting to mock os.walk so I'm going to post the
latest version of my bashateignore review and we can use .bashateignore to
both exclude as well as include the files to be processed using the
gitignore syntax.

Starting with the list of files in the repo is also an option, and
excluding from there...but I'm not going to have that tonight.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Robert Collins]

Well, git knows all the files in-tree, right? Or am I missing something here?

if-has-bash-hashbang-and-is-versioned-then-bashate-it?

-Rob

On 3 September 2014 13:26, Ian Wienand  wrote:
> On 09/02/2014 10:13 PM, Sean Dague wrote:
>>
>> One of the things that could make it better is to add file extensions to
>> all shell files in devstack. This would also solve the issue of gerrit
>> not syntax highlighting most of the files. If people are up for that,
>> I'll propose a rename patch to get us there. Then dumping the special
>> bashate discover bits is simple.
>
>
> I feel like adding .sh to bash to-be-sourced-only (library) files is
> probably a less common idiom.  It's just feeling, I don't think it's
> any sort of rule.
>
> So my first preference is for bashate to just punt the whole thing and
> only work on a list of files.  We can then discuss how best to match
> things in devstack so we don't add files but miss checking them.
>
> -i
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Ian Wienand]

On 09/02/2014 10:13 PM, Sean Dague wrote:

One of the things that could make it better is to add file extensions to
all shell files in devstack. This would also solve the issue of gerrit
not syntax highlighting most of the files. If people are up for that,
I'll propose a rename patch to get us there. Then dumping the special
bashate discover bits is simple.


I feel like adding .sh to bash to-be-sourced-only (library) files is
probably a less common idiom.  It's just feeling, I don't think it's
any sort of rule.

So my first preference is for bashate to just punt the whole thing and
only work on a list of files.  We can then discuss how best to match
things in devstack so we don't add files but miss checking them.

-i

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Sean Dague]

On 09/01/2014 01:38 AM, Ian Wienand wrote:
> On 08/29/2014 10:42 PM, Sean Dague wrote:
>> I'm actually kind of convinced now that none of these approaches are
>> what we need, and that we should instead have a .bashateignore file in
>> the root dir for the project instead, which would be regex that would
>> match files or directories to throw out of the walk.
> 
> Dean's idea of reading .gitignore might be good.
> 
> I had a quick poke at git dir.c:match_pathspec_item() and sort of came
> up with something similar [2] which roughly follows that and then only
> matches on files that have a shell-script mimetype; which I feel is
> probably sane for a default implementation.
> 
> IMO devstack should just generate it's own file-list to pass in for
> checking and bashate shouldn't have special guessing code for it
> 
> It all feels a bit like a solution looking for a problem.  Making
> bashate only work on a passed-in list of files and leaving generating
> those files up to the test infrastructure is probably would probably
> best the best KISS choice...
> 
> -i
> 
> [1] https://github.com/git/git/blob/master/dir.c#L216
> [2] https://review.openstack.org/#/c/117425/

Sure, I think part of this is we built the tool inside of devstack, and
now in the base case it's really awkward to use with devstack. I'd say
that's part of what I feel is the 0.x lifecycle for bashate, figure out
how to live as a separate tool and not add a ton of work to our existing
projects.

One of the things that could make it better is to add file extensions to
all shell files in devstack. This would also solve the issue of gerrit
not syntax highlighting most of the files. If people are up for that,
I'll propose a rename patch to get us there. Then dumping the special
bashate discover bits is simple.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Ian Wienand]

On 08/29/2014 10:42 PM, Sean Dague wrote:

I'm actually kind of convinced now that none of these approaches are
what we need, and that we should instead have a .bashateignore file in
the root dir for the project instead, which would be regex that would
match files or directories to throw out of the walk.


Dean's idea of reading .gitignore might be good.

I had a quick poke at git dir.c:match_pathspec_item() and sort of came
up with something similar [2] which roughly follows that and then only
matches on files that have a shell-script mimetype; which I feel is
probably sane for a default implementation.

IMO devstack should just generate it's own file-list to pass in for
checking and bashate shouldn't have special guessing code for it

It all feels a bit like a solution looking for a problem.  Making
bashate only work on a passed-in list of files and leaving generating
those files up to the test infrastructure is probably would probably
best the best KISS choice...

-i

[1] https://github.com/git/git/blob/master/dir.c#L216
[2] https://review.openstack.org/#/c/117425/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Dean Troyer]

On Fri, Aug 29, 2014 at 9:02 AM, Sean Dague  wrote:

> If pathspec did the right thing, pulling in the extra dep would be fine,
> but it doesn't seem like it does.
>

After looking at it with fresh eyes, the issue could be resolved by
combining two methods from pathspec and still leveraging the regex
compilation stuff, which is complicated...that part it gets right enough.

I think I got it worked out properly, and it might be even flexible enough
to replace discover_files() with the right patterns in .bashateignore.  If
not we can inject patterns too, I did that for .gitignore and
.bashateifnore. ;)

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Sean Dague]

On 08/29/2014 08:53 AM, Dean Troyer wrote:
> On Fri, Aug 29, 2014 at 7:42 AM, Sean Dague  > wrote:
> 
> Integrating bashate into something as complicated as devstack, the file
> ignore problem has come up.
> 
> We seem to have 3 approaches out under review right now:
> 
> https://review.openstack.org/#/c/117425 : --exclude-dirs
> https://review.openstack.org/#/c/115794 : --exclude-dirs (different
> implementation)
> https://review.openstack.org/#/c/113892 : removing hidden directories
> 
> I'm actually kind of convinced now that none of these approaches are
> what we need, and that we should instead have a .bashateignore file in
> the root dir for the project instead, which would be regex that would
> match files or directories to throw out of the walk.
> 
> I think that would handle the concerns that everyone is having, and
> hopefully provides a more clear set of semantics in integrating.
> 
> Anyone up for taking a stab at this patch?
> 
> 
> I started the other night and ran into the usual semantic problems wrt
> meaning...rather than re-invent this wheel I found the pathspec module
> another new dependency!) that purports to do .gitignore-style handling,
> only it doesn't.  It's closer to  rsync include file syntax.  I managed
> to get it really close only to fail on handling bare directories
> properly.  Example:
> 
> Ignoring a doc directory in .gitignore:
> doc
> 
> Ignoring a doc directory in my trial:
> doc/
> 
> It occurs to me that fixing this too means maybe I started down the
> wrong path.  This matters to be because I want to also leverage the
> existing .gitignore files we have.
> 
> Just to join the party I pushed up the working state
> in https://review.openstack.org/117772.

If pathspec did the right thing, pulling in the extra dep would be fine,
but it doesn't seem like it does.

What if we just used 'glob' instead, find all the glob patterns and
intersect them out? I think there is a little bit of trickiness around
directories, but as glob.glob('topleveldir') matches it, I think that
intersection probably would work out fine as well.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [bashate] .bashateignore

[Dean Troyer]

On Fri, Aug 29, 2014 at 7:42 AM, Sean Dague  wrote:

> Integrating bashate into something as complicated as devstack, the file
> ignore problem has come up.
>
> We seem to have 3 approaches out under review right now:
>
> https://review.openstack.org/#/c/117425 : --exclude-dirs
> https://review.openstack.org/#/c/115794 : --exclude-dirs (different
> implementation)
> https://review.openstack.org/#/c/113892 : removing hidden directories
>
> I'm actually kind of convinced now that none of these approaches are
> what we need, and that we should instead have a .bashateignore file in
> the root dir for the project instead, which would be regex that would
> match files or directories to throw out of the walk.
>
> I think that would handle the concerns that everyone is having, and
> hopefully provides a more clear set of semantics in integrating.
>
> Anyone up for taking a stab at this patch?
>

I started the other night and ran into the usual semantic problems wrt
meaning...rather than re-invent this wheel I found the pathspec module
another new dependency!) that purports to do .gitignore-style handling,
only it doesn't.  It's closer to  rsync include file syntax.  I managed to
get it really close only to fail on handling bare directories properly.
 Example:

Ignoring a doc directory in .gitignore:
doc

Ignoring a doc directory in my trial:
doc/

It occurs to me that fixing this too means maybe I started down the wrong
path.  This matters to be because I want to also leverage the existing
.gitignore files we have.

Just to join the party I pushed up the working state in
https://review.openstack.org/117772.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [bashate] .bashateignore

[Sean Dague]

Integrating bashate into something as complicated as devstack, the file
ignore problem has come up.

We seem to have 3 approaches out under review right now:

https://review.openstack.org/#/c/117425 : --exclude-dirs
https://review.openstack.org/#/c/115794 : --exclude-dirs (different
implementation)
https://review.openstack.org/#/c/113892 : removing hidden directories

I'm actually kind of convinced now that none of these approaches are
what we need, and that we should instead have a .bashateignore file in
the root dir for the project instead, which would be regex that would
match files or directories to throw out of the walk.

I think that would handle the concerns that everyone is having, and
hopefully provides a more clear set of semantics in integrating.

Anyone up for taking a stab at this patch?

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev