Re: resultsdb roadmap

[Róman Joost]

Hi Tim,

just a small ping :)

On Mon, Feb 16, 2015 at 09:17:50AM +1000, Róman Joost wrote:
  [...]
  This was less of a desired final outcome for us than it was the easiest
  way to get started.
  
  I think that our current method of having gigantic text-only log files
  is far from optimal (and something that we did years ago in AutoQA
  before making the logs more easily accessible) but we haven't improved
  the log accessibility/readability in taskotron yet.
 
 Good to know. I guess this is something to discuss and work on?

   3. Are there plans/ideas on implementing a waiver mechanism?
  
  Yeah, it'll have to happen before we start gating any
  rawhide/branched/stable builds or updates for Fedora. We don't have a
  design yet but it'll happen at some point.
 

If this thread has reached a dead end, I think the best way forward is
to come up with an implementation proposal for either of the two items:
keeping more test informatin in resultsdb or a waiver mechanism.

Kind Regards,
-- 
Róman Joost
Software Engineer, HSS - Software Engineering and Development (Brisbane)
email: rjo...@redhat.com | tz: UTC+10
irc: #hss #rpmdiff


pgpI95jSTNy30.pgp
Description: PGP signature
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: resultsdb roadmap

[Róman Joost]

Dear Kamil,

On Thu, Feb 12, 2015 at 03:41:38AM -0500, Kamil Paral wrote:
  [...]
  I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
  test result translates to an entry in Resultsdb as an outcome (PASSED,
  FAILED). 
 
 If you are interested, we support more outcome keywords, like INFO or 
 NEEDS_INSPECTION:
 https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library.html#module-libtaskotron.check

Great. Thanks for the pointer.

  Yet the question for the developer as to why it failed is
  important.
  
  I have mainly three main questions around Taskatron and specifically
  Resultsdb:
  
  1. Ed Santiago is currently running RPMGrill here:
  
  http://rpmgrill-fc20.edsantiago.com:5000/recent
  
which in terms of presenting results to the user provides a different
experience. The results are grouped by test and provide more insight
as to why a test failed. Test results in Resultsdb currently seem to
have only one outcome and technical details are left on the build
master.
  
How does a representation like RPMGrill translate into Resultsdb? Are
there plans/ideas on how to provide a more diversified result
representation in Resultsdb (e.g. error reason, hint on how to solve
an error).
 
 At the moment, each result contains a link to the task log, which you can 
 inspect, e.g.:
 https://taskotron.fedoraproject.org/resultsdb/results?job_id=45746
 points to
 https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/36226/steps/runtask/logs/stdio
 
 That log contains both taskotron execution info and above statements and task 
 output. If you know an undocumented trick, you can also strip down the path a 
 bit arrive to the buildbot job info page:
 https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/36226/
 and there you have access to taskotron.log, which contains full debug info, 
 which we use for debugging libtaskotron issues. This is usually not needed by 
 package maintainers, so we don't advertise this too much yet.

Thanks.

 I agree it would be great if taskotron could execute rpmgrill, and
 then upload the results to your special application and nicely display
 them. Or alternatively generate the html report locally. Unfortunately
 we don't have a support for storing and displaying artifacts (e.g. a
 html page) at the moment -- but we plan to do it in the future.
 
 If you want to have something working right now, I see several ways to
 go:
 
 1. You could print out a text representation of the rpmgrill results
 into the log. You could also post the results into your web
 applications, and then include a hyperlink in the log in order to
 point people to a better visualization.
 
 2. I guess we could change the Log → hyperlink in resultsdb to point
 to your web application directly, rather than into our buildbot log.
 However, I see two concerns with that, one is that accessing the task
 log itself would be non-trivial, and second that displaying any
 results at all would be relying on your web app availability. If it
 went down, nobody could see any results at all.
 
 In the future, generating the html result locally and then showing it
 from resultsdb interface is probably the way to go.

That's were we would like to help to make this happen. Before discussing
any technical details, my first e-mail was aimed to figure out if this
is a desirable feature or not. If that's something which looks like it
is part of where resultsdb is going, I could start a technical
discussion in another thread?

  3. Are there plans/ideas on implementing a waiver mechanism?
 
 We will definitely need this, for most of the checks. But we haven't
 started yet designing that feature, so it won't be available any time
 soon.
Thanks for sharing.

 
 BTW, I talked to Miroslav Vadkerti on DevConf about rpmgrill. We
 should meet up again in the following weeks and try to cook some
 actionable plan for rpmgrill in Taskotron.

Wow great!

Would it make sense to create a roadmap document or is there one
already?

Kind Regards,
-- 
Róman Joost
Software Engineer, HSS - Software Engineering and Development (Brisbane)
email: rjo...@redhat.com | tz: UTC+10
irc: #hss #rpmdiff


pgpKpKb_pbMVx.pgp
Description: PGP signature
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: resultsdb roadmap

[Róman Joost]

Dear Tim,

On Thu, Feb 12, 2015 at 10:31:23AM +0100, Tim Flink wrote:
  [...]
  I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
  test result translates to an entry in Resultsdb as an outcome (PASSED,
  FAILED). Yet the question for the developer as to why it failed is
  important.
 
 I don't think that desire is unique to rpmgrill and something that we
 decided to leave out of the initial Taskotron system.
 
 My initial thought is that that a discrete state (PASS, FAIL,
 NEEDS_INSPECTION etc.) a one-line-or-so summary and link to output
 from rpmgrill would work well. Are there other things that you'd like
 to see in terms of output?
From what Kamil pointed out, there is no state missing.

  1. Ed Santiago is currently running RPMGrill here:
  
  http://rpmgrill-fc20.edsantiago.com:5000/recent
  
which in terms of presenting results to the user provides a
  different experience. The results are grouped by test and provide
  more insight as to why a test failed. Test results in Resultsdb
  currently seem to have only one outcome and technical details are
  left on the build master.
 
 This was less of a desired final outcome for us than it was the easiest
 way to get started.
 
 I think that our current method of having gigantic text-only log files
 is far from optimal (and something that we did years ago in AutoQA
 before making the logs more easily accessible) but we haven't improved
 the log accessibility/readability in taskotron yet.

Good to know. I guess this is something to discuss and work on?


How does a representation like RPMGrill translate into Resultsdb?
  Are there plans/ideas on how to provide a more diversified result
representation in Resultsdb (e.g. error reason, hint on how to solve
an error).
 
 I think that having rpmgrill output static html would likely be the
 best approach here unless you're planning to continue supporting the
 webapp that you linked to (and from previous conversations, I'm under
 the impression that was never intended to be a long-term thing).

If we'd have a way to turn the TAP output into details retrievable from
resultsdb, it'll be better than rpmgrill printing HTML?


  2. Are there plans/ideas about implementing a (fulltext-) search over
results?
 
 Nothing written down, no but that's been on my wishlist since before
 Taskotron really got started. Unless there's more demand for the
 fulltext indexing than I think there is, we'd probably start with
 something less complicated like indexing the summary statements.
 
 I'm open to other ideas, though. Just concerned about what fulltext
 indexing of all logs might entail.

True. We've had quite big logs if some big packages contain a lot of
regressions. Good to know that it is on the wish list.

  3. Are there plans/ideas on implementing a waiver mechanism?
 
 Yeah, it'll have to happen before we start gating any
 rawhide/branched/stable builds or updates for Fedora. We don't have a
 design yet but it'll happen at some point.

Ok. This also seems like something we might be able to contribute to.
Perhaps better to discuss this in another thread?

  [1] - https://git.fedorahosted.org/git/rpmgrill.git
  
  PS: Yes - I'm aware that there is already a bitbucket repository of
  rpmgrill for Taskotron, but have only had a quick glimpse at it.
 
 Someone was planning to work on getting rpmgrill working in taskotron
 and that bitbucket repo was meant to hold the task that's executed in
 taskotron.
 
 From our end, that is being tracked as:
 
 https://phab.qadevel.cloud.fedoraproject.org/T281
 
 Let us know if you have any other questions or concerns.

Thanks. I'll have a look at the tasks. I don't want to make any
promises, but I hope we'll find some time to get the rpmgrill tasks in a
'done' state.

Thanks and Regards,
-- 
Róman Joost
Software Engineer, HSS - Software Engineering and Development (Brisbane)
email: rjo...@redhat.com | tz: UTC+10
irc: #hss #rpmdiff


pgpzRdvdn6kVs.pgp
Description: PGP signature
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: resultsdb roadmap

[Kamil Paral]

 Hi,
 
 I've already tried to ask on #fedora-qa, but I think the mailing list is
 the better medium to discuss this. Thanks to jskladen for answering.
 
 We've been looking into Taskotron and Resultsdb for a while and would
 like to deploy an instance running RPMGrill[1] and become contributors.
 
 I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
 test result translates to an entry in Resultsdb as an outcome (PASSED,
 FAILED). 

If you are interested, we support more outcome keywords, like INFO or 
NEEDS_INSPECTION:
https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library.html#module-libtaskotron.check

 Yet the question for the developer as to why it failed is
 important.
 
 I have mainly three main questions around Taskatron and specifically
 Resultsdb:
 
 1. Ed Santiago is currently running RPMGrill here:
 
 http://rpmgrill-fc20.edsantiago.com:5000/recent
 
   which in terms of presenting results to the user provides a different
   experience. The results are grouped by test and provide more insight
   as to why a test failed. Test results in Resultsdb currently seem to
   have only one outcome and technical details are left on the build
   master.
 
   How does a representation like RPMGrill translate into Resultsdb? Are
   there plans/ideas on how to provide a more diversified result
   representation in Resultsdb (e.g. error reason, hint on how to solve
   an error).

At the moment, each result contains a link to the task log, which you can 
inspect, e.g.:
https://taskotron.fedoraproject.org/resultsdb/results?job_id=45746
points to
https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/36226/steps/runtask/logs/stdio

That log contains both taskotron execution info and above statements and task 
output. If you know an undocumented trick, you can also strip down the path a 
bit arrive to the buildbot job info page:
https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/36226/
and there you have access to taskotron.log, which contains full debug info, 
which we use for debugging libtaskotron issues. This is usually not needed by 
package maintainers, so we don't advertise this too much yet.

I agree it would be great if taskotron could execute rpmgrill, and then upload 
the results to your special application and nicely display them. Or 
alternatively generate the html report locally. Unfortunately we don't have a 
support for storing and displaying artifacts (e.g. a html page) at the moment 
-- but we plan to do it in the future.

If you want to have something working right now, I see several ways to go:

1. You could print out a text representation of the rpmgrill results into the 
log. You could also post the results into your web applications, and then 
include a hyperlink in the log in order to point people to a better 
visualization.

2. I guess we could change the Log → hyperlink in resultsdb to point to your 
web application directly, rather than into our buildbot log. However, I see two 
concerns with that, one is that accessing the task log itself would be 
non-trivial, and second that displaying any results at all would be relying on 
your web app availability. If it went down, nobody could see any results at all.

In the future, generating the html result locally and then showing it from 
resultsdb interface is probably the way to go.

 
 2. Are there plans/ideas about implementing a (fulltext-) search over
   results?

I'll let others to respond to this, I have no idea.

 
 3. Are there plans/ideas on implementing a waiver mechanism?

We will definitely need this, for most of the checks. But we haven't started 
yet designing that feature, so it won't be available any time soon.


BTW, I talked to Miroslav Vadkerti on DevConf about rpmgrill. We should meet up 
again in the following weeks and try to cook some actionable plan for rpmgrill 
in Taskotron.

Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

Re: resultsdb roadmap

[Tim Flink]

On Mon, 9 Feb 2015 15:48:45 +1000
Róman Joost rjo...@redhat.com wrote:

 Hi,
 
 I've already tried to ask on #fedora-qa, but I think the mailing list
 is the better medium to discuss this. Thanks to jskladen for
 answering.
 
 We've been looking into Taskotron and Resultsdb for a while and would
 like to deploy an instance running RPMGrill[1] and become
 contributors.
 
 I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
 test result translates to an entry in Resultsdb as an outcome (PASSED,
 FAILED). Yet the question for the developer as to why it failed is
 important.

I don't think that desire is unique to rpmgrill and something that we
decided to leave out of the initial Taskotron system.

My initial thought is that that a discrete state (PASS, FAIL,
NEEDS_INSPECTION etc.) a one-line-or-so summary and link to output
from rpmgrill would work well. Are there other things that you'd like
to see in terms of output?

 I have mainly three main questions around Taskatron and specifically
 Resultsdb:
 
 1. Ed Santiago is currently running RPMGrill here:
 
 http://rpmgrill-fc20.edsantiago.com:5000/recent
 
   which in terms of presenting results to the user provides a
 different experience. The results are grouped by test and provide
 more insight as to why a test failed. Test results in Resultsdb
 currently seem to have only one outcome and technical details are
 left on the build master.

This was less of a desired final outcome for us than it was the easiest
way to get started.

I think that our current method of having gigantic text-only log files
is far from optimal (and something that we did years ago in AutoQA
before making the logs more easily accessible) but we haven't improved
the log accessibility/readability in taskotron yet.

   How does a representation like RPMGrill translate into Resultsdb?
 Are there plans/ideas on how to provide a more diversified result
   representation in Resultsdb (e.g. error reason, hint on how to solve
   an error).

I think that having rpmgrill output static html would likely be the
best approach here unless you're planning to continue supporting the
webapp that you linked to (and from previous conversations, I'm under
the impression that was never intended to be a long-term thing).

 2. Are there plans/ideas about implementing a (fulltext-) search over
   results?

Nothing written down, no but that's been on my wishlist since before
Taskotron really got started. Unless there's more demand for the
fulltext indexing than I think there is, we'd probably start with
something less complicated like indexing the summary statements.

I'm open to other ideas, though. Just concerned about what fulltext
indexing of all logs might entail.

 3. Are there plans/ideas on implementing a waiver mechanism?

Yeah, it'll have to happen before we start gating any
rawhide/branched/stable builds or updates for Fedora. We don't have a
design yet but it'll happen at some point.


 [1] - https://git.fedorahosted.org/git/rpmgrill.git
 
 PS: Yes - I'm aware that there is already a bitbucket repository of
 rpmgrill for Taskotron, but have only had a quick glimpse at it.

Someone was planning to work on getting rpmgrill working in taskotron
and that bitbucket repo was meant to hold the task that's executed in
taskotron.

From our end, that is being tracked as:

https://phab.qadevel.cloud.fedoraproject.org/T281

Let us know if you have any other questions or concerns.

Tim
 Thanks and Kind Regards,



pgp1gn8KbhsVb.pgp
Description: OpenPGP digital signature
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel

resultsdb roadmap

[Róman Joost]

Hi,

I've already tried to ask on #fedora-qa, but I think the mailing list is
the better medium to discuss this. Thanks to jskladen for answering.

We've been looking into Taskotron and Resultsdb for a while and would
like to deploy an instance running RPMGrill[1] and become contributors.

I assume, running rpmgrill in Taskotron could mean, that each rpmgrill
test result translates to an entry in Resultsdb as an outcome (PASSED,
FAILED). Yet the question for the developer as to why it failed is
important.

I have mainly three main questions around Taskatron and specifically
Resultsdb:

1. Ed Santiago is currently running RPMGrill here:

http://rpmgrill-fc20.edsantiago.com:5000/recent

  which in terms of presenting results to the user provides a different
  experience. The results are grouped by test and provide more insight
  as to why a test failed. Test results in Resultsdb currently seem to
  have only one outcome and technical details are left on the build
  master.

  How does a representation like RPMGrill translate into Resultsdb? Are
  there plans/ideas on how to provide a more diversified result
  representation in Resultsdb (e.g. error reason, hint on how to solve
  an error).

2. Are there plans/ideas about implementing a (fulltext-) search over
  results?

3. Are there plans/ideas on implementing a waiver mechanism?

[1] - https://git.fedorahosted.org/git/rpmgrill.git

PS: Yes - I'm aware that there is already a bitbucket repository of
rpmgrill for Taskotron, but have only had a quick glimpse at it.

Thanks and Kind Regards,
-- 
Róman Joost
Software Engineer, HSS - Software Engineering and Development (Brisbane)
email: rjo...@redhat.com | tz: UTC+10
irc: #hss #rpmdiff


pgpQiXYEo6dU9.pgp
Description: PGP signature
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/qa-devel