Re: [openstack-dev] [bashate] .bashateignore

2014-09-02 Thread Ian Wienand

On 09/02/2014 10:13 PM, Sean Dague wrote:

One of the things that could make it better is to add file extensions to
all shell files in devstack. This would also solve the issue of gerrit
not syntax highlighting most of the files. If people are up for that,
I'll propose a rename patch to get us there. Then dumping the special
bashate discover bits is simple.


I feel like adding .sh to bash to-be-sourced-only (library) files is
probably a less common idiom.  It's just feeling, I don't think it's
any sort of rule.

So my first preference is for bashate to just punt the whole thing and
only work on a list of files.  We can then discuss how best to match
things in devstack so we don't add files but miss checking them.

-i

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-09-02 Thread Robert Collins
Well, git knows all the files in-tree, right? Or am I missing something here?

if-has-bash-hashbang-and-is-versioned-then-bashate-it?

-Rob

On 3 September 2014 13:26, Ian Wienand iwien...@redhat.com wrote:
 On 09/02/2014 10:13 PM, Sean Dague wrote:

 One of the things that could make it better is to add file extensions to
 all shell files in devstack. This would also solve the issue of gerrit
 not syntax highlighting most of the files. If people are up for that,
 I'll propose a rename patch to get us there. Then dumping the special
 bashate discover bits is simple.


 I feel like adding .sh to bash to-be-sourced-only (library) files is
 probably a less common idiom.  It's just feeling, I don't think it's
 any sort of rule.

 So my first preference is for bashate to just punt the whole thing and
 only work on a list of files.  We can then discuss how best to match
 things in devstack so we don't add files but miss checking them.

 -i


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-09-02 Thread Dean Troyer
On Tue, Sep 2, 2014 at 8:32 PM, Robert Collins robe...@robertcollins.net
wrote:

 Well, git knows all the files in-tree, right? Or am I missing something
 here?

 if-has-bash-hashbang-and-is-versioned-then-bashate-it?


It's not quote that simple, none of the include files have a shebang line;
I've always felt that having that in an include file is an indication that
the file is (also) a stand-alone script.   Shocco (the docs processor)
wants one too.

I think I've given up attempting to mock os.walk so I'm going to post the
latest version of my bashateignore review and we can use .bashateignore to
both exclude as well as include the files to be processed using the
gitignore syntax.

Starting with the list of files in the repo is also an option, and
excluding from there...but I'm not going to have that tonight.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-09-02 Thread Ian Wienand

On 09/03/2014 11:32 AM, Robert Collins wrote:

if-has-bash-hashbang-and-is-versioned-then-bashate-it?


That misses library files that aren't execed and have no #!

This might be an appropriate rule for test infrastructure to generate a 
list for their particular project, but IMO I don't think we need to 
start building that logic into bashate


-i

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-08-31 Thread Ian Wienand

On 08/29/2014 10:42 PM, Sean Dague wrote:

I'm actually kind of convinced now that none of these approaches are
what we need, and that we should instead have a .bashateignore file in
the root dir for the project instead, which would be regex that would
match files or directories to throw out of the walk.


Dean's idea of reading .gitignore might be good.

I had a quick poke at git dir.c:match_pathspec_item() and sort of came
up with something similar [2] which roughly follows that and then only
matches on files that have a shell-script mimetype; which I feel is
probably sane for a default implementation.

IMO devstack should just generate it's own file-list to pass in for
checking and bashate shouldn't have special guessing code for it

It all feels a bit like a solution looking for a problem.  Making
bashate only work on a passed-in list of files and leaving generating
those files up to the test infrastructure is probably would probably
best the best KISS choice...

-i

[1] https://github.com/git/git/blob/master/dir.c#L216
[2] https://review.openstack.org/#/c/117425/

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-08-29 Thread Dean Troyer
On Fri, Aug 29, 2014 at 7:42 AM, Sean Dague s...@dague.net wrote:

 Integrating bashate into something as complicated as devstack, the file
 ignore problem has come up.

 We seem to have 3 approaches out under review right now:

 https://review.openstack.org/#/c/117425 : --exclude-dirs
 https://review.openstack.org/#/c/115794 : --exclude-dirs (different
 implementation)
 https://review.openstack.org/#/c/113892 : removing hidden directories

 I'm actually kind of convinced now that none of these approaches are
 what we need, and that we should instead have a .bashateignore file in
 the root dir for the project instead, which would be regex that would
 match files or directories to throw out of the walk.

 I think that would handle the concerns that everyone is having, and
 hopefully provides a more clear set of semantics in integrating.

 Anyone up for taking a stab at this patch?


I started the other night and ran into the usual semantic problems wrt
meaning...rather than re-invent this wheel I found the pathspec module
another new dependency!) that purports to do .gitignore-style handling,
only it doesn't.  It's closer to  rsync include file syntax.  I managed to
get it really close only to fail on handling bare directories properly.
 Example:

Ignoring a doc directory in .gitignore:
doc

Ignoring a doc directory in my trial:
doc/

It occurs to me that fixing this too means maybe I started down the wrong
path.  This matters to be because I want to also leverage the existing
.gitignore files we have.

Just to join the party I pushed up the working state in
https://review.openstack.org/117772.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-08-29 Thread Sean Dague
On 08/29/2014 08:53 AM, Dean Troyer wrote:
 On Fri, Aug 29, 2014 at 7:42 AM, Sean Dague s...@dague.net
 mailto:s...@dague.net wrote:
 
 Integrating bashate into something as complicated as devstack, the file
 ignore problem has come up.
 
 We seem to have 3 approaches out under review right now:
 
 https://review.openstack.org/#/c/117425 : --exclude-dirs
 https://review.openstack.org/#/c/115794 : --exclude-dirs (different
 implementation)
 https://review.openstack.org/#/c/113892 : removing hidden directories
 
 I'm actually kind of convinced now that none of these approaches are
 what we need, and that we should instead have a .bashateignore file in
 the root dir for the project instead, which would be regex that would
 match files or directories to throw out of the walk.
 
 I think that would handle the concerns that everyone is having, and
 hopefully provides a more clear set of semantics in integrating.
 
 Anyone up for taking a stab at this patch?
 
 
 I started the other night and ran into the usual semantic problems wrt
 meaning...rather than re-invent this wheel I found the pathspec module
 another new dependency!) that purports to do .gitignore-style handling,
 only it doesn't.  It's closer to  rsync include file syntax.  I managed
 to get it really close only to fail on handling bare directories
 properly.  Example:
 
 Ignoring a doc directory in .gitignore:
 doc
 
 Ignoring a doc directory in my trial:
 doc/
 
 It occurs to me that fixing this too means maybe I started down the
 wrong path.  This matters to be because I want to also leverage the
 existing .gitignore files we have.
 
 Just to join the party I pushed up the working state
 in https://review.openstack.org/117772.

If pathspec did the right thing, pulling in the extra dep would be fine,
but it doesn't seem like it does.

What if we just used 'glob' instead, find all the glob patterns and
intersect them out? I think there is a little bit of trickiness around
directories, but as glob.glob('topleveldir') matches it, I think that
intersection probably would work out fine as well.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [bashate] .bashateignore

2014-08-29 Thread Dean Troyer
On Fri, Aug 29, 2014 at 9:02 AM, Sean Dague s...@dague.net wrote:

 If pathspec did the right thing, pulling in the extra dep would be fine,
 but it doesn't seem like it does.


After looking at it with fresh eyes, the issue could be resolved by
combining two methods from pathspec and still leveraging the regex
compilation stuff, which is complicated...that part it gets right enough.

I think I got it worked out properly, and it might be even flexible enough
to replace discover_files() with the right patterns in .bashateignore.  If
not we can inject patterns too, I did that for .gitignore and
.bashateifnore. ;)

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev