Re: [openstack-dev] [bashate] .bashateignore
On 09/02/2014 10:13 PM, Sean Dague wrote: One of the things that could make it better is to add file extensions to all shell files in devstack. This would also solve the issue of gerrit not syntax highlighting most of the files. If people are up for that, I'll propose a rename patch to get us there. Then dumping the special bashate discover bits is simple. I feel like adding .sh to bash to-be-sourced-only (library) files is probably a less common idiom. It's just feeling, I don't think it's any sort of rule. So my first preference is for bashate to just punt the whole thing and only work on a list of files. We can then discuss how best to match things in devstack so we don't add files but miss checking them. -i ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
Well, git knows all the files in-tree, right? Or am I missing something here? if-has-bash-hashbang-and-is-versioned-then-bashate-it? -Rob On 3 September 2014 13:26, Ian Wienand iwien...@redhat.com wrote: On 09/02/2014 10:13 PM, Sean Dague wrote: One of the things that could make it better is to add file extensions to all shell files in devstack. This would also solve the issue of gerrit not syntax highlighting most of the files. If people are up for that, I'll propose a rename patch to get us there. Then dumping the special bashate discover bits is simple. I feel like adding .sh to bash to-be-sourced-only (library) files is probably a less common idiom. It's just feeling, I don't think it's any sort of rule. So my first preference is for bashate to just punt the whole thing and only work on a list of files. We can then discuss how best to match things in devstack so we don't add files but miss checking them. -i ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
On Tue, Sep 2, 2014 at 8:32 PM, Robert Collins robe...@robertcollins.net wrote: Well, git knows all the files in-tree, right? Or am I missing something here? if-has-bash-hashbang-and-is-versioned-then-bashate-it? It's not quote that simple, none of the include files have a shebang line; I've always felt that having that in an include file is an indication that the file is (also) a stand-alone script. Shocco (the docs processor) wants one too. I think I've given up attempting to mock os.walk so I'm going to post the latest version of my bashateignore review and we can use .bashateignore to both exclude as well as include the files to be processed using the gitignore syntax. Starting with the list of files in the repo is also an option, and excluding from there...but I'm not going to have that tonight. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
On 09/03/2014 11:32 AM, Robert Collins wrote: if-has-bash-hashbang-and-is-versioned-then-bashate-it? That misses library files that aren't execed and have no #! This might be an appropriate rule for test infrastructure to generate a list for their particular project, but IMO I don't think we need to start building that logic into bashate -i ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
On 08/29/2014 10:42 PM, Sean Dague wrote: I'm actually kind of convinced now that none of these approaches are what we need, and that we should instead have a .bashateignore file in the root dir for the project instead, which would be regex that would match files or directories to throw out of the walk. Dean's idea of reading .gitignore might be good. I had a quick poke at git dir.c:match_pathspec_item() and sort of came up with something similar [2] which roughly follows that and then only matches on files that have a shell-script mimetype; which I feel is probably sane for a default implementation. IMO devstack should just generate it's own file-list to pass in for checking and bashate shouldn't have special guessing code for it It all feels a bit like a solution looking for a problem. Making bashate only work on a passed-in list of files and leaving generating those files up to the test infrastructure is probably would probably best the best KISS choice... -i [1] https://github.com/git/git/blob/master/dir.c#L216 [2] https://review.openstack.org/#/c/117425/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [bashate] .bashateignore
Integrating bashate into something as complicated as devstack, the file ignore problem has come up. We seem to have 3 approaches out under review right now: https://review.openstack.org/#/c/117425 : --exclude-dirs https://review.openstack.org/#/c/115794 : --exclude-dirs (different implementation) https://review.openstack.org/#/c/113892 : removing hidden directories I'm actually kind of convinced now that none of these approaches are what we need, and that we should instead have a .bashateignore file in the root dir for the project instead, which would be regex that would match files or directories to throw out of the walk. I think that would handle the concerns that everyone is having, and hopefully provides a more clear set of semantics in integrating. Anyone up for taking a stab at this patch? -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
On Fri, Aug 29, 2014 at 7:42 AM, Sean Dague s...@dague.net wrote: Integrating bashate into something as complicated as devstack, the file ignore problem has come up. We seem to have 3 approaches out under review right now: https://review.openstack.org/#/c/117425 : --exclude-dirs https://review.openstack.org/#/c/115794 : --exclude-dirs (different implementation) https://review.openstack.org/#/c/113892 : removing hidden directories I'm actually kind of convinced now that none of these approaches are what we need, and that we should instead have a .bashateignore file in the root dir for the project instead, which would be regex that would match files or directories to throw out of the walk. I think that would handle the concerns that everyone is having, and hopefully provides a more clear set of semantics in integrating. Anyone up for taking a stab at this patch? I started the other night and ran into the usual semantic problems wrt meaning...rather than re-invent this wheel I found the pathspec module another new dependency!) that purports to do .gitignore-style handling, only it doesn't. It's closer to rsync include file syntax. I managed to get it really close only to fail on handling bare directories properly. Example: Ignoring a doc directory in .gitignore: doc Ignoring a doc directory in my trial: doc/ It occurs to me that fixing this too means maybe I started down the wrong path. This matters to be because I want to also leverage the existing .gitignore files we have. Just to join the party I pushed up the working state in https://review.openstack.org/117772. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
On 08/29/2014 08:53 AM, Dean Troyer wrote:
On Fri, Aug 29, 2014 at 7:42 AM, Sean Dague s...@dague.net
mailto:s...@dague.net wrote:
Integrating bashate into something as complicated as devstack, the file
ignore problem has come up.
We seem to have 3 approaches out under review right now:
https://review.openstack.org/#/c/117425 : --exclude-dirs
https://review.openstack.org/#/c/115794 : --exclude-dirs (different
implementation)
https://review.openstack.org/#/c/113892 : removing hidden directories
I'm actually kind of convinced now that none of these approaches are
what we need, and that we should instead have a .bashateignore file in
the root dir for the project instead, which would be regex that would
match files or directories to throw out of the walk.
I think that would handle the concerns that everyone is having, and
hopefully provides a more clear set of semantics in integrating.
Anyone up for taking a stab at this patch?
I started the other night and ran into the usual semantic problems wrt
meaning...rather than re-invent this wheel I found the pathspec module
another new dependency!) that purports to do .gitignore-style handling,
only it doesn't. It's closer to rsync include file syntax. I managed
to get it really close only to fail on handling bare directories
properly. Example:
Ignoring a doc directory in .gitignore:
doc
Ignoring a doc directory in my trial:
doc/
It occurs to me that fixing this too means maybe I started down the
wrong path. This matters to be because I want to also leverage the
existing .gitignore files we have.
Just to join the party I pushed up the working state
in https://review.openstack.org/117772.
If pathspec did the right thing, pulling in the extra dep would be fine,
but it doesn't seem like it does.
What if we just used 'glob' instead, find all the glob patterns and
intersect them out? I think there is a little bit of trickiness around
directories, but as glob.glob('topleveldir') matches it, I think that
intersection probably would work out fine as well.
-Sean
--
Sean Dague
http://dague.net
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [bashate] .bashateignore
On Fri, Aug 29, 2014 at 9:02 AM, Sean Dague s...@dague.net wrote: If pathspec did the right thing, pulling in the extra dep would be fine, but it doesn't seem like it does. After looking at it with fresh eyes, the issue could be resolved by combining two methods from pathspec and still leveraging the regex compilation stuff, which is complicated...that part it gets right enough. I think I got it worked out properly, and it might be even flexible enough to replace discover_files() with the right patterns in .bashateignore. If not we can inject patterns too, I did that for .gitignore and .bashateifnore. ;) dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
