Re: How to distinct errors and failures in tests

[Nick N]

The problem is I don't actually use pure unittest, I have a custom runner
on top of the unittest.

The requested feature in the subject.


On Sat, Apr 24, 2021, 00:35 Jonathan Rogers 
wrote:

> On 4/23/21 5:26 PM, Nick N wrote:
> > Thank you Jonathan!
> >
> > Sure, pytest is more advanced framework and provide more features. I
> will definitely rewrite tests to pytest when my boss give me time. Though
> it doesn't provide requested feature.
>
> What prevents you from using pytest with your existing tests? It is
> designed to run unittest-style tests without any changes. Which requested
> feature do you need?
>
> https://docs.pytest.org/en/latest/how-to/unittest.html
>
> --
> Jonathan Rogers
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/ku3MQxMvfl8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/9c081724-4635-702c-b604-5decd39ff188%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CACA70iAT-kc000mAxo7HxPsyuMzuSxXsatr2EqnXpaW4CdQVZA%40mail.gmail.com.

Re: How to distinct errors and failures in tests

[Jonathan Rogers]

On 4/23/21 5:26 PM, Nick N wrote:

Thank you Jonathan!

Sure, pytest is more advanced framework and provide more features. I will 
definitely rewrite tests to pytest when my boss give me time. Though it doesn't 
provide requested feature.


What prevents you from using pytest with your existing tests? It is designed to 
run unittest-style tests without any changes. Which requested feature do you 
need?

https://docs.pytest.org/en/latest/how-to/unittest.html

--
Jonathan Rogers

--
You received this message because you are subscribed to the Google Groups "Jenkins 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/9c081724-4635-702c-b604-5decd39ff188%40gmail.com.

Re: How to distinct errors and failures in tests

[Nick N]

Thank you Jonathan!

Sure, pytest is more advanced framework and provide more features. I will
definitely rewrite tests to pytest when my boss give me time. Though it
doesn't provide requested feature.

By the way unless there is better solution I've implemented a workaround.
My framework just writes the empty file if there were no errors, so I have
to request 1 file to be sure all other exist.

On Fri, Apr 23, 2021, 22:44 JonathanRRogers 
wrote:

> I suggest you start using a test runner which can produce JUnit/XUnit
> output. I've used both nose and pytest, both of which can run unittest
> tests and produce JUnit-style XML output.
>
> On Wednesday, April 21, 2021 at 8:59:36 AM UTC-4 nikola...@gmail.com
> wrote:
>
>> Thank you for the answer. If I don't mistake, JUnit plugin just publishes
>> the report, but unittest can't produce JUint-compatible xml. Anyway, we
>> agree to consider builds with failed tests like failed. Tests generate some
>> artifacts, and I want to know all artifacts exist before trying to download
>> them.
>>
>> среда, 21 апреля 2021 г. в 15:03:35 UTC+3, ullrich...@gmail.com:
>>
>>> Typically you do not set the status of a build by setting the return
>>> code. Your build should never alter the return code just because the tests
>>> are failing. So I would suggest that your build always returns 0 if the
>>> build is successful (ignoring the test results). The JUnit plugin will do
>>> the build status handling for you: if there are test failures, then the
>>> JUnit plugin will set the build status to unstable. If not, the status
>>> remains successful. You can then navigate with lastSuccessful or lastStable
>>> to the last build without testfailures.
>>>
>>> BTW: I don’t think that there is a way to differentiate between test
>>> errors and failures in Jenkins yet.
>>>
>>> Am 21.04.2021 um 11:26 schrieb Nick N :
>>>
>>> Hey, guys!
>>>
>>> I have a freestyle project. It builds an app and run tests (python,
>>> unittest). It is quite straightforward if return codes are 0 then build is
>>> successful, otherwise it is failed. Earlier I returned amount of errors and
>>> failures as a returncode to see tests results on the dashboard (passed or
>>> failed). But test results may contain failures and errors, and I want to
>>> know if there were errors on tests or they all finished, but found some
>>> bugs. Actually I need to get last build (via jenkins api) without errors,
>>> no matter if there were failures.
>>>
>>> I've tried to look at "lastStableBuild", but it seems it sticks to 
>>> "lastSuccessfulBuild".
>>> I've tried to find information about build statuses but had no success.
>>> Particularly what are criteria of each of them? Maybe it is completely
>>> wrong approach and there's a better way?
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Jenkins Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to jenkinsci-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jenkinsci-users/CACA70iDR0X9RTWv2051vTuO3pC5odS10kssRYLQe6%2B1A2DjGDg%40mail.gmail.com
>>> 
>>> .
>>>
>>>
>>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/ku3MQxMvfl8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/a4a3b64e-d718-44b8-992a-55b4bae0159bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/CACA70iDAoNLu27ZzWzn5x34C-9KYbSoEcQWpoEeQjvuROz1_vg%40mail.gmail.com.

Re: How to distinct errors and failures in tests

[JonathanRRogers]

I suggest you start using a test runner which can produce JUnit/XUnit 
output. I've used both nose and pytest, both of which can run unittest 
tests and produce JUnit-style XML output.

On Wednesday, April 21, 2021 at 8:59:36 AM UTC-4 nikola...@gmail.com wrote:

> Thank you for the answer. If I don't mistake, JUnit plugin just publishes 
> the report, but unittest can't produce JUint-compatible xml. Anyway, we 
> agree to consider builds with failed tests like failed. Tests generate some 
> artifacts, and I want to know all artifacts exist before trying to download 
> them.
>
> среда, 21 апреля 2021 г. в 15:03:35 UTC+3, ullrich...@gmail.com: 
>
>> Typically you do not set the status of a build by setting the return 
>> code. Your build should never alter the return code just because the tests 
>> are failing. So I would suggest that your build always returns 0 if the 
>> build is successful (ignoring the test results). The JUnit plugin will do 
>> the build status handling for you: if there are test failures, then the 
>> JUnit plugin will set the build status to unstable. If not, the status 
>> remains successful. You can then navigate with lastSuccessful or lastStable 
>> to the last build without testfailures.
>>
>> BTW: I don’t think that there is a way to differentiate between test 
>> errors and failures in Jenkins yet.
>>
>> Am 21.04.2021 um 11:26 schrieb Nick N :
>>
>> Hey, guys!
>>
>> I have a freestyle project. It builds an app and run tests (python, 
>> unittest). It is quite straightforward if return codes are 0 then build is 
>> successful, otherwise it is failed. Earlier I returned amount of errors and 
>> failures as a returncode to see tests results on the dashboard (passed or 
>> failed). But test results may contain failures and errors, and I want to 
>> know if there were errors on tests or they all finished, but found some 
>> bugs. Actually I need to get last build (via jenkins api) without errors, 
>> no matter if there were failures.
>>
>> I've tried to look at "lastStableBuild", but it seems it sticks to 
>> "lastSuccessfulBuild". 
>> I've tried to find information about build statuses but had no success. 
>> Particularly what are criteria of each of them? Maybe it is completely 
>> wrong approach and there's a better way?
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Jenkins Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkinsci-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/jenkinsci-users/CACA70iDR0X9RTWv2051vTuO3pC5odS10kssRYLQe6%2B1A2DjGDg%40mail.gmail.com
>>  
>> 
>> .
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/a4a3b64e-d718-44b8-992a-55b4bae0159bn%40googlegroups.com.

Re: How to distinct errors and failures in tests

[nikola...@gmail.com]

Thank you for the answer. If I don't mistake, JUnit plugin just publishes 
the report, but unittest can't produce JUint-compatible xml. Anyway, we 
agree to consider builds with failed tests like failed. Tests generate some 
artifacts, and I want to know all artifacts exist before trying to download 
them.

среда, 21 апреля 2021 г. в 15:03:35 UTC+3, ullrich...@gmail.com: 

> Typically you do not set the status of a build by setting the return code. 
> Your build should never alter the return code just because the tests are 
> failing. So I would suggest that your build always returns 0 if the build 
> is successful (ignoring the test results). The JUnit plugin will do the 
> build status handling for you: if there are test failures, then the JUnit 
> plugin will set the build status to unstable. If not, the status remains 
> successful. You can then navigate with lastSuccessful or lastStable to the 
> last build without testfailures.
>
> BTW: I don’t think that there is a way to differentiate between test 
> errors and failures in Jenkins yet.
>
> Am 21.04.2021 um 11:26 schrieb Nick N :
>
> Hey, guys!
>
> I have a freestyle project. It builds an app and run tests (python, 
> unittest). It is quite straightforward if return codes are 0 then build is 
> successful, otherwise it is failed. Earlier I returned amount of errors and 
> failures as a returncode to see tests results on the dashboard (passed or 
> failed). But test results may contain failures and errors, and I want to 
> know if there were errors on tests or they all finished, but found some 
> bugs. Actually I need to get last build (via jenkins api) without errors, 
> no matter if there were failures.
>
> I've tried to look at "lastStableBuild", but it seems it sticks to 
> "lastSuccessfulBuild". 
> I've tried to find information about build statuses but had no success. 
> Particularly what are criteria of each of them? Maybe it is completely 
> wrong approach and there's a better way?
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-users/CACA70iDR0X9RTWv2051vTuO3pC5odS10kssRYLQe6%2B1A2DjGDg%40mail.gmail.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/0d1f597e-ae2d-44d6-b53f-85217e5e1ec7n%40googlegroups.com.

Re: How to distinct errors and failures in tests

[Ullrich Hafner]

Typically you do not set the status of a build by setting the return code. Your 
build should never alter the return code just because the tests are failing. So 
I would suggest that your build always returns 0 if the build is successful 
(ignoring the test results). The JUnit plugin will do the build status handling 
for you: if there are test failures, then the JUnit plugin will set the build 
status to unstable. If not, the status remains successful. You can then 
navigate with lastSuccessful or lastStable to the last build without 
testfailures.

BTW: I don’t think that there is a way to differentiate between test errors and 
failures in Jenkins yet.

> Am 21.04.2021 um 11:26 schrieb Nick N :
> 
> Hey, guys!
> 
> I have a freestyle project. It builds an app and run tests (python, 
> unittest). It is quite straightforward if return codes are 0 then build is 
> successful, otherwise it is failed. Earlier I returned amount of errors and 
> failures as a returncode to see tests results on the dashboard (passed or 
> failed). But test results may contain failures and errors, and I want to know 
> if there were errors on tests or they all finished, but found some bugs. 
> Actually I need to get last build (via jenkins api) without errors, no matter 
> if there were failures.
> 
> I've tried to look at "lastStableBuild", but it seems it sticks to 
> "lastSuccessfulBuild". I've tried to find information about build statuses 
> but had no success. Particularly what are criteria of each of them? Maybe it 
> is completely wrong approach and there's a better way?
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-users+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-users/CACA70iDR0X9RTWv2051vTuO3pC5odS10kssRYLQe6%2B1A2DjGDg%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/61D82A4F-D2EF-4D49-909C-D0D55BC4A556%40gmail.com.