Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Matthias Kuhn]

On 03/08/2017 11:28 AM, Alessandro Pasotti wrote:

> I tried to reproduce the issue locally without success.
> 
> I suggest for now to either split the printing tests into a separate
> file or better (if there is single test failing) just decorate the test
> as allowed to fail so that the whole suite does not fail.
> 
> When done, we can re-enable the server test in travis.

There is no `AllowFailure` available IIRC. There is only an
`ExpectedFailure` which will fail if the test succeeds.

Adding an infrastructure for something like this would be much
appreciated (similar to how ExpectedFailure is handled).
But splitting into a different file will be easier.

Matthias
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Alessandro Pasotti]

On Tue, Mar 7, 2017 at 9:12 AM, Paul Blottiere 
wrote:

>
> > It sounds strange to me that it was never observed in production, but I
> did not search the hub for an existing ticket, it might have been reported.
>
>
> The test seems to fail in very specific conditions (that I have not yet
> found). As Mathias suggested, I made a loop on the test. And even with 1000
> iterations, it's still not failing whatever the branch (master or 2.14)...
> Maybe it has never been observed.
>
>
> Paul
>
>
>
I tried to reproduce the issue locally without success.

I suggest for now to either split the printing tests into a separate file
or better (if there is single test failing) just decorate the test as
allowed to fail so that the whole suite does not fail.

When done, we can re-enable the server test in travis.



> On 03/07/2017 08:53 AM, Alessandro Pasotti wrote:
>
> On Tue, Mar 7, 2017 at 8:48 AM, Paul Blottiere <
> paul.blotti...@oslandia.com> wrote:
>
>> Alessandro,
>>
>>
>> > do you have clue if the multi-threading rendering could be the root
>> cause of this issue?
>>
>> The test is failing on travis for 2.14 too. However, PR about the
>> replacement of the QgsMapRenderer and the parallel rendering activation are
>> not backported in 2.14. So, as it happens both on master and 2.14, it seems
>> to be independent from the rendering step.
>>
>>
>> Only unit tests on the getprint service has been backported and before
>> that, there was none. So I think that the failing test is just underlying
>> an old issue which was there even before we started to refactor the server.
>>
>>
>> What do you think?
>>
>
> I'm afraid you are right ...
>
> It will be difficult to spot it out, by the way I think that the test did
> a good work to discover this issue.
>
> It sounds strange to me that it was never observed in production, but I
> did not search the hub for an existing ticket, it might have been reported.
>
>
>
>> Paul
>>
>>
>>
>> On 03/07/2017 08:16 AM, Alessandro Pasotti wrote:
>>
>> On Mon, Mar 6, 2017 at 5:30 PM, Paul Blottiere 
>> wrote:
>>
>>> Hi Alessandro,
>>>
>>>
>>> can you please have a look?
>>>
>>> I'm afraid this failure is not a problem in the test itself but a
>>> symptom of more serious issue: the map is completely missing from the
>>> generated print.
>>>
>>> FYI, the same test also fails on 2.14 as discussed on the following
>>> commit:
>>>
>>>
>>> I took a look but I cannot reproduce the failing test
>>> test_wms_getprint_srs.
>>>
>>> Actually, I have tested it within the next environment without issues:
>>> - master on Debian
>>> - release-2_14 on Ubuntu.
>>>
>>> Can anyone confirm?
>>>
>>> Paul
>>>
>>
>>
>> Paul,
>>
>> Thanks for looking into this, I'll do some local testing shortly, do you
>> have clue if the multi-threading rendering could be the root cause of this
>> issue?
>>
>> A s a side note, I would recommend to refactor the server test and split
>> it in smaller dedicated tests for the different services.
>>
>>
>>
>> --
>> Alessandro Pasotti
>> w3:   www.itopen.it
>>
>>
>> ___
>> Qgis-developer mailing listqgis-develo...@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>> --
> Alessandro Pasotti w3:   www.itopen.it
>
>


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Paul Blottiere]

> It sounds strange to me that it was never observed in production, but
I did not search the hub for an existing ticket, it might have been
reported.


The test seems to fail in very specific conditions (that I have not yet
found). As Mathias suggested, I made a loop on the test. And even with
1000 iterations, it's still not failing whatever the branch (master or
2.14)... Maybe it has never been observed.


Paul


On 03/07/2017 08:53 AM, Alessandro Pasotti wrote:
> On Tue, Mar 7, 2017 at 8:48 AM, Paul Blottiere
> > wrote:
>
> Alessandro,
>
>
> > do you have clue if the multi-threading rendering could be the root 
> cause of this issue?
>
> The test is failing on travis for 2.14 too. However, PR about the
> replacement of the QgsMapRenderer and the parallel rendering
> activation are not backported in 2.14. So, as it happens both on
> master and 2.14, it seems to be independent from the rendering step.
>
>
> Only unit tests on the getprint service has been backported and
> before that, there was none. So I think that the failing test is
> just underlying an old issue which was there even before we
> started to refactor the server.
>
>
> What do you think?
>
>
> I'm afraid you are right ...
>
> It will be difficult to spot it out, by the way I think that the test
> did a good work to discover this issue.
>
> It sounds strange to me that it was never observed in production, but
> I did not search the hub for an existing ticket, it might have been
> reported.
>
>
>
> Paul
>
>
>
> On 03/07/2017 08:16 AM, Alessandro Pasotti wrote:
>> On Mon, Mar 6, 2017 at 5:30 PM, Paul Blottiere
>> > wrote:
>>
>> Hi Alessandro,
>>
>>
 can you please have a look?

 I'm afraid this failure is not a problem in the test itself
 but a symptom of more serious issue: the map is completely
 missing from the generated print.
>>> FYI, the same test also fails on 2.14 as discussed on the
>>> following commit:
>>
>> I took a look but I cannot reproduce the failing test
>> test_wms_getprint_srs.
>>
>> Actually, I have tested it within the next environment
>> without issues:
>> - master on Debian
>> - release-2_14 on Ubuntu.
>>
>> Can anyone confirm?
>>
>> Paul
>>
>>
>>
>> Paul,
>>
>> Thanks for looking into this, I'll do some local testing shortly,
>> do you have clue if the multi-threading rendering could be the
>> root cause of this issue?
>>
>> A s a side note, I would recommend to refactor the server test
>> and split it in smaller dedicated tests for the different services.
>>
>>
>>
>> -- 
>> Alessandro Pasotti
>> w3:   www.itopen.it 
>>
>>
>> ___
>> Qgis-developer mailing list
>> Qgis-developer@lists.osgeo.org
>> 
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> 
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> 
>
> -- 
> Alessandro Pasotti w3:   www.itopen.it 
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Alessandro Pasotti]

On Tue, Mar 7, 2017 at 8:48 AM, Paul Blottiere 
wrote:

> Alessandro,
>
>
> > do you have clue if the multi-threading rendering could be the root
> cause of this issue?
>
> The test is failing on travis for 2.14 too. However, PR about the
> replacement of the QgsMapRenderer and the parallel rendering activation are
> not backported in 2.14. So, as it happens both on master and 2.14, it seems
> to be independent from the rendering step.
>
>
> Only unit tests on the getprint service has been backported and before
> that, there was none. So I think that the failing test is just underlying
> an old issue which was there even before we started to refactor the server.
>
>
> What do you think?
>

I'm afraid you are right ...

It will be difficult to spot it out, by the way I think that the test did a
good work to discover this issue.

It sounds strange to me that it was never observed in production, but I did
not search the hub for an existing ticket, it might have been reported.



> Paul
>
>
>
> On 03/07/2017 08:16 AM, Alessandro Pasotti wrote:
>
> On Mon, Mar 6, 2017 at 5:30 PM, Paul Blottiere 
> wrote:
>
>> Hi Alessandro,
>>
>>
>> can you please have a look?
>>
>> I'm afraid this failure is not a problem in the test itself but a symptom
>> of more serious issue: the map is completely missing from the generated
>> print.
>>
>> FYI, the same test also fails on 2.14 as discussed on the following
>> commit:
>>
>>
>> I took a look but I cannot reproduce the failing test
>> test_wms_getprint_srs.
>>
>> Actually, I have tested it within the next environment without issues:
>> - master on Debian
>> - release-2_14 on Ubuntu.
>>
>> Can anyone confirm?
>>
>> Paul
>>
>
>
> Paul,
>
> Thanks for looking into this, I'll do some local testing shortly, do you
> have clue if the multi-threading rendering could be the root cause of this
> issue?
>
> A s a side note, I would recommend to refactor the server test and split
> it in smaller dedicated tests for the different services.
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
>
> ___
> Qgis-developer mailing listqgis-develo...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
>


-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Paul Blottiere]

Alessandro,


> do you have clue if the multi-threading rendering could be the root
cause of this issue?

The test is failing on travis for 2.14 too. However, PR about the
replacement of the QgsMapRenderer and the parallel rendering activation
are not backported in 2.14. So, as it happens both on master and 2.14,
it seems to be independent from the rendering step.


Only unit tests on the getprint service has been backported and before
that, there was none. So I think that the failing test is just
underlying an old issue which was there even before we started to
refactor the server.


What do you think?


Paul



On 03/07/2017 08:16 AM, Alessandro Pasotti wrote:
> On Mon, Mar 6, 2017 at 5:30 PM, Paul Blottiere
> > wrote:
>
> Hi Alessandro,
>
>
>>> can you please have a look?
>>>
>>> I'm afraid this failure is not a problem in the test itself but
>>> a symptom of more serious issue: the map is completely missing
>>> from the generated print.
>> FYI, the same test also fails on 2.14 as discussed on the
>> following commit:
>
> I took a look but I cannot reproduce the failing test
> test_wms_getprint_srs.
>
> Actually, I have tested it within the next environment without issues:
> - master on Debian
> - release-2_14 on Ubuntu.
>
> Can anyone confirm?
>
> Paul
>
>
>
> Paul,
>
> Thanks for looking into this, I'll do some local testing shortly, do
> you have clue if the multi-threading rendering could be the root cause
> of this issue?
>
> A s a side note, I would recommend to refactor the server test and
> split it in smaller dedicated tests for the different services.
>
>
>
> -- 
> Alessandro Pasotti
> w3:   www.itopen.it 
>
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Alessandro Pasotti]

On Mon, Mar 6, 2017 at 5:30 PM, Paul Blottiere 
wrote:

> Hi Alessandro,
>
>
> can you please have a look?
>
> I'm afraid this failure is not a problem in the test itself but a symptom
> of more serious issue: the map is completely missing from the generated
> print.
>
> FYI, the same test also fails on 2.14 as discussed on the following commit:
>
>
> I took a look but I cannot reproduce the failing test
> test_wms_getprint_srs.
>
> Actually, I have tested it within the next environment without issues:
> - master on Debian
> - release-2_14 on Ubuntu.
>
> Can anyone confirm?
>
> Paul
>


Paul,

Thanks for looking into this, I'll do some local testing shortly, do you
have clue if the multi-threading rendering could be the root cause of this
issue?

A s a side note, I would recommend to refactor the server test and split it
in smaller dedicated tests for the different services.



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Matthias Kuhn]

Hi Paul,

On 03/06/2017 05:30 PM, Paul Blottiere wrote:
> Hi Alessandro,
> 
> 
>>> can you please have a look?
>>>
>>> I'm afraid this failure is not a problem in the test itself but a
>>> symptom of more serious issue: the map is completely missing from the
>>> generated print.
>> FYI, the same test also fails on 2.14 as discussed on the following
>> commit:
> 
> I took a look but I cannot reproduce the failing test
> test_wms_getprint_srs.
> 
> Actually, I have tested it within the next environment without issues:
> - master on Debian
> - release-2_14 on Ubuntu.

The test also only failed one out of 30 times on travis, so I guess it's
a tricky one to get a lead.

I'd recommend to run it in a loop locally and see if you can reproduce
it this way.

If you really don't find a way to reproduce this locally but have some
suspicion, you can also introduce some additional tests or debug output
that might help to assess the problem and re-enable the test on travis.

This way at least we have more information on travis the next time it fails.

Bests
Matthias
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Matthias Kuhn]

On 04/03/2017 09:18, Alessandro Pasotti wrote:

> Hi Paul,
>
> can you please have a look?
>
> I'm afraid this failure is not a problem in the test itself but a
> symptom of more serious issue: the map is completely missing from the
> generated print.
FYI, the same test also fails on 2.14 as discussed on the following commit:

https://github.com/qgis/QGIS/commit/d9f9c9df54c50c28f240dae22cc6876f10e3d865#commitcomment-21011556

Thanks
Matthias

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Alessandro Pasotti]

On Sat, Mar 4, 2017 at 8:25 AM, Matthias Kuhn  wrote:

> Following the conversation from
> https://github.com/qgis/QGIS/commit/89d4998 , this is an announcement
> to notify that the PyQgsServerTest :: test_wms_getprint_srs  has been
> disabled.
>
> Work has been underway to remove "noisy" tests from Travis which cause
> false alarms and these tests regularly fail for unrelated changes.
> False alarms are bad as they undermine confidence in the CI results
> and encourage contributors to ignore the test results.
>
> If you're interested in seeing this test continue as part of the CI
> infrastructure here's an example of the test failing on an unrelated
> commit:
>
> https://travis-ci.org/qgis/QGIS/jobs/207575956
>
> Matthias
>
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer



Hi Paul,

can you please have a look?

I'm afraid this failure is not a problem in the test itself but a symptom
of more serious issue: the map is completely missing from the generated
print.

The test was introduced by this commit.
https://github.com/qgis/QGIS/commit/f3a5c98178103afc5da3d7d5d3bc2df6930cb9bc#diff-260b51384025856fac2dc9682481a865R569

Check it with:
https://travis-ci.org/qgis/QGIS/jobs/207575956#L1477

Thanks!

-- 
Alessandro Pasotti
w3:   www.itopen.it
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] PyQgsServerTest :: test_wms_getprint_srs disabled

[Matthias Kuhn]

Following the conversation from
https://github.com/qgis/QGIS/commit/89d4998 , this is an announcement
to notify that the PyQgsServerTest :: test_wms_getprint_srs  has been
disabled.

Work has been underway to remove "noisy" tests from Travis which cause
false alarms and these tests regularly fail for unrelated changes.
False alarms are bad as they undermine confidence in the CI results
and encourage contributors to ignore the test results.

If you're interested in seeing this test continue as part of the CI
infrastructure here's an example of the test failing on an unrelated
commit:

https://travis-ci.org/qgis/QGIS/jobs/207575956

Matthias 

___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer