We have changed the resolution of junit5 engines 2 times.
Now I have got the result with 3.0.0-M5. I think it is what you expect:
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @
surefire-no-build-fail ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.jeffjensen.surefire.ApplicationTest
19:43:18.789 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating CacheAwareContextLoaderDelegate from class
[org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
19:43:18.821 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating BootstrapContext using constructor [public
org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
19:43:18.910 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating TestContextBootstrapper for test class
[com.jeffjensen.surefire.ApplicationTest] from class
[org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
19:43:18.944 [main] INFO
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
- Neither @ContextConfiguration nor @ContextHierarchy found for test class
[com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
19:43:18.951 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
19:43:18.952 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
19:43:18.952 [main] INFO
org.springframework.test.context.support.AbstractContextLoader - Could not
detect default resource locations for test class
[com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
{-context.xml, Context.groovy}.
19:43:18.953 [main] INFO
org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
- Could not detect default configuration classes for test class
[com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
any static, non-private, non-final, nested classes annotated with
@Configuration.
19:43:19.024 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating CacheAwareContextLoaderDelegate from class
[org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
19:43:19.024 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating BootstrapContext using constructor [public
org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
19:43:19.025 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating TestContextBootstrapper for test class
[com.jeffjensen.surefire.ApplicationTest] from class
[org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
19:43:19.026 [main] INFO
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
- Neither @ContextConfiguration nor @ContextHierarchy found for test class
[com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
19:43:19.027 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
19:43:19.028 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
19:43:19.029 [main] INFO
org.springframework.test.context.support.AbstractContextLoader - Could not
detect default resource locations for test class
[com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
{-context.xml, Context.groovy}.
19:43:19.029 [main] INFO
org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
- Could not detect default configuration classes for test class
[com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
any static, non-private, non-final, nested classes annotated with
@Configuration.
19:43:19.035 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating CacheAwareContextLoaderDelegate from class
[org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate]
19:43:19.035 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating BootstrapContext using constructor [public
org.springframework.test.context.support.DefaultBootstrapContext(java.lang.Class,org.springframework.test.context.CacheAwareContextLoaderDelegate)]
19:43:19.037 [main] DEBUG org.springframework.test.context.BootstrapUtils -
Instantiating TestContextBootstrapper for test class
[com.jeffjensen.surefire.ApplicationTest] from class
[org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper]
19:43:19.038 [main] INFO
org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTestContextBootstrapper
- Neither @ContextConfiguration nor @ContextHierarchy found for test class
[com.jeffjensen.surefire.ApplicationTest], using SpringBootContextLoader
19:43:19.040 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTest-context.xml] does not exist
19:43:19.040 [main] DEBUG
org.springframework.test.context.support.AbstractContextLoader - Did not
detect default resource location for test class
[com.jeffjensen.surefire.ApplicationTest]: class path resource
[com/jeffjensen/surefire/ApplicationTestContext.groovy] does not exist
19:43:19.040 [main] INFO
org.springframework.test.context.support.AbstractContextLoader - Could not
detect default resource locations for test class
[com.jeffjensen.surefire.ApplicationTest]: no resource found for suffixes
{-context.xml, Context.groovy}.
19:43:19.041 [main] INFO
org.springframework.test.context.support.AnnotationConfigContextLoaderUtils
- Could not detect default configuration classes for test class
[com.jeffjensen.surefire.ApplicationTest]: ApplicationTest does not declare
any static, non-private, non-final, nested classes annotated with
@Configuration.
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0.417 s <<< FAILURE! - in com.jeffjensen.surefire.ApplicationTest
[ERROR] com.jeffjensen.surefire.ApplicationTest Time elapsed: 0.417 s <<<
ERROR!
java.lang.NoClassDefFoundError:
org/springframework/web/bind/annotation/ControllerAdvice
Caused by: java.lang.ClassNotFoundException:
org.springframework.web.bind.annotation.ControllerAdvice
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ApplicationTest » NoClassDefFound
org/springframework/web/bind/annotation/Cont...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 6.723 s
[INFO] Finished at: 2020-06-13T19:43:19+02:00
[INFO]
------------------------------------------------------------------------
On Sat, Jun 13, 2020 at 6:56 PM Jeff Jensen <jjen...@apache.org> wrote:
> This reproduces the issue:
> https://github.com/jeffjensen/maven-surefire-3.0.0-m4-no-build-fail
>
> Please let me know if I can help further.
>
>
> On Sat, Jun 13, 2020 at 9:09 AM Tibor Digaňa
> <tibor.dig...@googlemail.com.invalid> wrote:
>
> > yes, the reproducible project would be great to have.
> > Thx
> >
> > On Sat, Jun 13, 2020 at 4:04 PM Jeff Jensen <jjen...@apache.org> wrote:
> >
> > > Thank you Enrico and Tibor for verifying it is a probable new issue.
> > >
> > > I cannot upload this project (customer product/private repo) but will
> > make
> > > a project that reproduces it and reply again.
> > >
> > >
> > > On Sat, Jun 13, 2020 at 8:24 AM Enrico Olivelli <eolive...@gmail.com>
> > > wrote:
> > >
> > > > Il Sab 13 Giu 2020, 15:07 Jeff Jensen <jjen...@apache.org> ha
> scritto:
> > > >
> > > > > Hi Enrico,
> > > > > Thanks for the reply.
> > > > >
> > > > > Surefire 2.22.2 correctly fails the build. This is with Java 8 and
> > > JUnit
> > > > > 5.
> > > > >
> > > >
> > > >
> > > > This sounds interesting.
> > > >
> > > > It looks like a big bug to me and it is worth a JIRA
> > > >
> > > > It would help a lot if you could attach a simple reproducer for the
> > > > problem.
> > > > If you know surefire codebase it would also help a lot to create the
> > > > reproducer as an integration test
> > > >
> > > > Thanks
> > > >
> > > > Enrico
> > > >
> > > >
> > > > >
> > > > > On Sat, Jun 13, 2020 at 12:41 AM Enrico Olivelli <
> > eolive...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Jeff
> > > > > >
> > > > > > Il Sab 13 Giu 2020, 00:15 Jeff Jensen <
> > > > jeffjen...@upstairstechnology.com
> > > > > >
> > > > > > ha scritto:
> > > > > >
> > > > > > > I looked for this issue in JIRA but haven't found anything yet.
> > > > Anyone
> > > > > > > know if this has been reported and/or fixed?
> > > > > > >
> > > > > > > Our scenario is a failure occurs at startup of tests but
> Surefire
> > > > > doesn't
> > > > > > > fail the build. Specifically, Spring controller tests aren't
> > > failing
> > > > > > when
> > > > > > > there is a Spring configuration problem.
> > > > > > >
> > > > > > > Spring top-level thrown exception is:
> > > > > > > java.lang.IllegalStateException: Failed to load
> > > ApplicationContext
> > > > > > >
> > > > > > > but Surefire doesn't report fail. Surefire output results is:
> > > > > > > Tests run: 0, Failures: 0, Errors: 0, Skipped: 0,
> > > > > > >
> > > > > > > Running the tests in an IDE correctly fail for the setup
> problem.
> > > > > > >
> > > > > >
> > > > > > Which IDE?
> > > > > >
> > > > > > Did you try other versions of surefire? Like the latest from 2.x
> > > > release
> > > > > > line?
> > > > > >
> > > > > > Regards
> > > > > > Enrico
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Cheers
> > Tibor
> >
>
