Hey Bruno, Yeah, the SQL style syntax is exactly the sort of thing I am wishing I could apply to Job properties and other entities within Jenkins as well. Since we need Folders for RBAC anyway, Folders end up being our single and dominant form of metadata and we use them to define product based job groups. That works okay, but in particular, its a poor mechanism for fetching the list of all products that went to production this past Thursday, which on the surface of it, would be a perfectly normal question to ask a build manager.
As I see it, there are three types of settings/metadata I would want to track.... - Config settings established prior to a job run: what repositories, what security context, what Docker image, which type of build-step constructors are being used, what slave pool should be used, what test environments should be used, who gets notified, etc.... - Settings that are purely metadata: what project name do I want to apply to these groups of Jenkins items, what are the sorts of stage/environment names I want to use across projects to track comparative progress between projects... - Results determined by the job execution: What artifacts were made, what compute units actually did the build, how long did it take, did it succeed, where did the artifacts go... For the Config settings, it seems helpful to me that these setting be robustly accessible prior to the job running, as one of the key things one might want to examine is how the planned process ends up comparing to the factual results. These being computers, of course they will do what you tell them, but you might have open pieces in your job-chain that someone else can edit, or your rules may be so abstract and complicated comparing the expected to the actual is often a very useful visualization. For the purely metadata, I would hope that they would be minimal, as it is often difficult to know in advance the ways you will want to sort and group your information later. And for the results set, it really feels to me that the Jenkins Job centered approach might not be right here. It might be better to look first at the artifacts, which are obviously related to the job, but are the product of it, and thus slightly different. Have you used our new CloudBees Analytics plugins at all? They are meant to do some of this results and post-process settings sorting. Thanks Bruno, Always a pleasure. Gus On Mon, Mar 30, 2015 at 12:48 PM, 'Bruno P. Kinoshita' via Jenkins Developers <[email protected]> wrote: > Hey Gus > > > -- Does anyone else attempt to make queries against Jenkins of this sort? > > For scientists and researchers it's extremely important to be able to > store, retrieve and search metadata. We have an idea of how to extend the > metadata plug-in [1] (probably via several Pull Requests), so that it > provides the following features: > > - Automatically collect metadata of Jenkins items (jobs, builds, > artifacts, etc) > - Let users collect extra metadata with builders/publishers/etc. For > instance, Cell Profiler [2] has a metadata module that can collect metadata > from file names. So a file named TRIAGE-CP001-20151120035003.txt could > generate metadata such as {code: "CP001", date:"2015-11-20 03:50:03UTC"} if > the user selected a File Name Metadata Collector > - Store the metadata in an embedded triple store (Jena) using an > ontology created for Jenkins, which will import other ontologies such as > DC, W3C Prov, etc. > - A search module and interface, that supports both Lucene (via > jena-text module) and SPARQL. > > The idea behind it is to have a pre defined ontology, which will ease > learning about the data model used for the metadata, and transform Jenkins > into a SPARQL endpoint. That way you'll be able to even query remote > Jenkins for metadata too. > > We have had some discussions about it already, but I will try to have a > working version in the following months. As an example, this question "Show > me all the jobs that use this set of slaves" could be answered by a SPARQL > query similar to this one (syntax not tested, just a quick example): > > PREFIX jenkins: http:// > SELECT ?jobs > WHERE { > ?jobs a jenkins:Job . > ?jobs jenkins:hasSlave ?slave . > ?slave jenkins:slaveName ?slaveName . > ?slaveName IN ('...') > } > > > -- Has anyone used the Metadata plugin, or some other plugin or > home-grown solution to solve this issue? > > I used it and had a look at the code, and I feel like I can extend it to > support the ontology and SPARQL. @Ioannis used it too, maybe he can give > his feedback on it too. > > This is something that is necessary for experiments and scientific > pipelines. It would be interesting if we could in some way join forces to > work on it :-) > > Cheers > > [1] https://groups.google.com/forum/#!topic/biouno-developers/2u1dQ0uX5RM > [2] http://www.cellprofiler.org/CPmanual/Metadata.html > > ------------------------------ > *From:* Gus Reiber <[email protected]> > *To:* [email protected] > *Sent:* Monday, March 30, 2015 1:29 PM > *Subject:* Grouping jobs by project, status, or attributes other than > folder location > > Hey all, > I currently using the folder plugin to group my projects together, but > am finding that it is difficult to get cross-cutting job > status/stage/attribute views of the jobs contained within those folders. > Here are a set of examples of the sorts of queries I would like to be able > to generate views based upon: > > - Show me all the jobs related to a particular project (for me, this > is typically the folder) > - Show me all the jobs that deploy to "production" (many folders might > have a job that does this) > - Show me all the jobs that use this set of slaves > - Show me all the jobs triggered by a submission from 'John Dowe' (I > can never trust that J.D., he is all over the place) > - Show me all the jobs tied to this repository > > The metadata plugin seems like it is geared towards this issue: > https://wiki.jenkins-ci.org/display/JENKINS/Metadata+plugin > > ....so I guess I have two questions: > -- Does anyone else attempt to make queries against Jenkins of this sort? > -- Has anyone used the Metadata plugin, or some other plugin or home-grown > solution to solve this issue? > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/3f9c0f9e-abb0-4092-804e-489f505be2ff%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-dev/3f9c0f9e-abb0-4092-804e-489f505be2ff%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Developers" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-dev/3ndjfF1otus/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/2095356288.690480.1427744927153.JavaMail.yahoo%40mail.yahoo.com > <https://groups.google.com/d/msgid/jenkinsci-dev/2095356288.690480.1427744927153.JavaMail.yahoo%40mail.yahoo.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAOcHHXwgD13RegkE9o0K53AzpL5o2cFaYj6_eRVB0Pee1vRsiw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
