Re: [Avocado-devel] correlating xunit output to job-results
Dne 28. 11. 18 v 15:35 Brian J. Murrell napsal(a):
> On Mon, 2018-11-26 at 17:03 +0100, Lukáš Doktor wrote:
>>
>> This is a new information. So are you actually looking for job--
>> MM-DDTHH.MM-UID only, or are you looking for test-name => test-
>> output-dir mapping?
>
> I'm looking to correlate a given display of one of mnay xunit test
> results in Jenkins job to the one of many job--MM-DDTHH.MM-UID
> directories I have collected for the test run.
>
> My xunit output shows me I have a failed test. I want to go to the
> job--MM-DDTHH.MM-UID I have in the archives for the job to see more
> information about it.
>
I see, so I actually misunderstood you. This seems even simpler, how about
this: https://github.com/avocado-framework/avocado/pull/2922
>> My idea was to basically do something like:
>>
>> for name in xunit.gettests().get('name'):
>> tid = name.split('-')[0]
>> print(glob.glob("latest/test-results/%s-*" % tid))
>>
>> which although not nice should always return only 1 match.
>
> This doesn't correlate to an avocado/job-results/job--MM-DDTHH.MM-
> UID directory does it though?
>
>>
>>> Looking at a given Test instance I can see lots of attributes and
>>> actually now that I look, I can see several attributes that embed
>>> that
>>> job--MM-DDTHH.MM-UID value into pathnames such as _stderr_file,
>>> _stdout_file, _Test__logfile, _Test__sysinfodir, _Test__logdir,
>>> _ssh_logfile, _Test__outputdir.
>>>
>>> I can surely peel the value I am looking for out of any one of
>>> these
>>> but of course, that feels like I am peeking under the kimono and
>>> using
>>> something that is subject to change in the future.
>
> So while I can do this, I can't seem to display it in any way such that
> it gets into the xunit results in any way. I can print it, in
> __init__(), say, but that doesn't end up in the xunit results file.
>
> Hrm. Looking again, I see that self.log.info() results end up on the
> xunit Standard Output. This is probably where I can add this. I just
> need to convert all tests to using my subclass and calling the parent's
> setUp() in their own setUp(), etc.
>
That seems like an ugly hack... Hopefully we'll find a more suitable solution
:-)
>> I think you understand well (based on the first response). Basically
>> I run a job, it produces several hundreds of MB of log files, which
>> are attached as artifacts and then several tenths of MB is attached
>> again as the XUNIT file, because the "debug.log" is always embedded
>> into the XUNIT results. It'd be nice to simply add link to the
>> archived results, but I haven't found a way back then...
>
> You could just delete the duplicates before archiving them in Jenkins.
> I do that for the jobs-results/*/html/ subdirs since the links in them
> are broken anyway.
>
Well, it's just a symlink, not taking much space... It's there for
compatibility reasons with 36lts and had been removed after 52lts was
released...
Regards,
Lukáš
> Or maybe have avocado not produce the debug.log if xunit output is
> enabled.
>
> Cheers,
> b.
>
signature.asc
Description: OpenPGP digital signature
Re: [Avocado-devel] correlating xunit output to job-results
On Mon, 2018-11-26 at 17:03 +0100, Lukáš Doktor wrote:
>
> This is a new information. So are you actually looking for job--
> MM-DDTHH.MM-UID only, or are you looking for test-name => test-
> output-dir mapping?
I'm looking to correlate a given display of one of mnay xunit test
results in Jenkins job to the one of many job--MM-DDTHH.MM-UID
directories I have collected for the test run.
My xunit output shows me I have a failed test. I want to go to the
job--MM-DDTHH.MM-UID I have in the archives for the job to see more
information about it.
> My idea was to basically do something like:
>
> for name in xunit.gettests().get('name'):
> tid = name.split('-')[0]
> print(glob.glob("latest/test-results/%s-*" % tid))
>
> which although not nice should always return only 1 match.
This doesn't correlate to an avocado/job-results/job--MM-DDTHH.MM-
UID directory does it though?
>
> > Looking at a given Test instance I can see lots of attributes and
> > actually now that I look, I can see several attributes that embed
> > that
> > job--MM-DDTHH.MM-UID value into pathnames such as _stderr_file,
> > _stdout_file, _Test__logfile, _Test__sysinfodir, _Test__logdir,
> > _ssh_logfile, _Test__outputdir.
> >
> > I can surely peel the value I am looking for out of any one of
> > these
> > but of course, that feels like I am peeking under the kimono and
> > using
> > something that is subject to change in the future.
So while I can do this, I can't seem to display it in any way such that
it gets into the xunit results in any way. I can print it, in
__init__(), say, but that doesn't end up in the xunit results file.
Hrm. Looking again, I see that self.log.info() results end up on the
xunit Standard Output. This is probably where I can add this. I just
need to convert all tests to using my subclass and calling the parent's
setUp() in their own setUp(), etc.
> I think you understand well (based on the first response). Basically
> I run a job, it produces several hundreds of MB of log files, which
> are attached as artifacts and then several tenths of MB is attached
> again as the XUNIT file, because the "debug.log" is always embedded
> into the XUNIT results. It'd be nice to simply add link to the
> archived results, but I haven't found a way back then...
You could just delete the duplicates before archiving them in Jenkins.
I do that for the jobs-results/*/html/ subdirs since the links in them
are broken anyway.
Or maybe have avocado not produce the debug.log if xunit output is
enabled.
Cheers,
b.
signature.asc
Description: This is a digitally signed message part
Re: [Avocado-devel] correlating xunit output to job-results
Dne 26. 11. 18 v 16:24 Brian J. Murrell napsal(a):
> On Mon, 2018-11-26 at 13:47 +0100, Lukáš Doktor wrote:
>>
>> Hello Brian,
>>
>> the results and xunit results are independent, but at the time of
>> creating the results we should have all necessary information
>> available.
>
> Indeed. I was thinking the same.
>
>> The question is how to embed the links.
>
> I don't even care to have actual links. I just want to see a reference
> to a job--MM-DDTHH.MM-UID in the xunit output somewhere.
>
This is a new information. So are you actually looking for
job--MM-DDTHH.MM-UID only, or are you looking for test-name =>
test-output-dir mapping? Because first I understood the second is the main goal
but from this note I think you are looking for the job-id and job-dir-name
instead.
>> The main problem I see is that the xunit results can be stored
>> somewhere else than in the standard location and (very often) the
>> results are moved as well.
>
> Indeed, hence my not even wanting links.
>
>> For that reason absolute path doesn't make much sense to me and
>> relative path could be troublesome as well.
>
> Agreed. I'd say forget the links.
>
>> Do you have a setup where such information would make sense?
>
> Jenkins. We import the xunit results into Jenkins which then displays
> them nicely formatted in "Error Message", "Stacktrace", and "Standard
> Output" sections. We also then add the job--MM-DDTHH.MM-UID dirs
> as artifacts.
>
Sure, that's the common way :-)
>> If not and you are only looking for a way to map these, you can use
>> the serialized test id. The test-reference (and therefor the test
>> results dir) is strictly defined as "$testId-$testName:$testVariant",
>> therefor when you split the name (or path) by `-` you'll get test-id
>> which is guaranteed to be unique across the job.
>
> Where/how are any of these referrable?
>
My idea was to basically do something like:
for name in xunit.gettests().get('name'):
tid = name.split('-')[0]
print(glob.glob("latest/test-results/%s-*" % tid))
which although not nice should always return only 1 match.
> Looking at a given Test instance I can see lots of attributes and
> actually now that I look, I can see several attributes that embed that
> job--MM-DDTHH.MM-UID value into pathnames such as _stderr_file,
> _stdout_file, _Test__logfile, _Test__sysinfodir, _Test__logdir,
> _ssh_logfile, _Test__outputdir.
>
> I can surely peel the value I am looking for out of any one of these
> but of course, that feels like I am peeking under the kimono and using
> something that is subject to change in the future.
>
The best way is `json` results, which is free-form, avocado machine-readable
results. We keep it as stable as possible (I know of one bigger change with the
test-id standardization and we kept +1 LTS overlap with backward compatibility).
>> PS: If you know of a nice way to let Jenkins read those files from
>> artifacts instead of embedding them, do let me know.
>
> I'm not entirely sure what you mean. Are you looking for Jenkins to
> provide some sort of nice clickable output format for those job--
> MM-DDTHH.MM-UID dirs other than just being able to traverse them as a
> "folder" in it's artifacts browser? If so, I know of no way better.
>
Yes and I looked for such way when started using jenkins, but at least at that
time there was no such way. I mean I can add link to the output, but that does
not sound right... Anyway good to know you don't see a better way either.
>> I'd be really interested (as currently I basically store the output
>> twice, which is not really efficient and I have to use `--xunit-max-
>> test-log-chars` to avoid really long outputs...)
>
> I must not be understanding what you are doing and what you are looking
> for as an alternative. Maybe if you have a link to an example.
>
I think you understand well (based on the first response). Basically I run a
job, it produces several hundreds of MB of log files, which are attached as
artifacts and then several tenths of MB is attached again as the XUNIT file,
because the "debug.log" is always embedded into the XUNIT results. It'd be nice
to simply add link to the archived results, but I haven't found a way back
then...
Regards,
Lukáš
> Cheers,
> b.
>
signature.asc
Description: OpenPGP digital signature
Re: [Avocado-devel] correlating xunit output to job-results
Dne 25. 11. 18 v 16:08 Brian J. Murrell napsal(a): > So, at the end of a test run of a thousand or so tests I have all of > this xunit output and all of these jobs-results directories. > > I can't seem to find any way to correlate one with the other. There is > no indication (that I have found) in the xunit file of those names that > are in the numerous job-results files. > > Is this a gap of some sort or is there something I am missing? > > Cheers, > b. > Hello Brian, the results and xunit results are independent, but at the time of creating the results we should have all necessary information available. The question is how to embed the links. The main problem I see is that the xunit results can be stored somewhere else than in the standard location and (very often) the results are moved as well. For that reason absolute path doesn't make much sense to me and relative path could be troublesome as well. Do you have a setup where such information would make sense? And ideally a draft how this information could be embed into XML file? If not and you are only looking for a way to map these, you can use the serialized test id. The test-reference (and therefor the test results dir) is strictly defined as "$testId-$testName:$testVariant", therefor when you split the name (or path) by `-` you'll get test-id which is guaranteed to be unique across the job. Note the current implementation uses serialized test-ids, but this is not mandatory. It's just one of the possible implementations... Regards, Lukáš PS: If you know of a nice way to let Jenkins read those files from artifacts instead of embedding them, do let me know. I'd be really interested (as currently I basically store the output twice, which is not really efficient and I have to use `--xunit-max-test-log-chars` to avoid really long outputs...) signature.asc Description: OpenPGP digital signature
