Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-16 Thread Andrea Aime 
And here we go:

https://github.com/geoserver/geoserver/pull/2599

With those changes I can build GeoServer locally within 6 minutes.
The two big speedups have been:

   - Splitting restconfig service into their own separate modules
   - Moving the gs-main security tests into their own separate module (this
   one alone cuts 50 seconds away from the build of both gs-main and the
   overall build time)

In the end I did not touch the app-schema tests as with the new setup they
are finishing just a few seconds after gs-web-gwc. Currently the "slow"
part is wfs->wms->gwc, to make it faster one would need to untangle wms
from its wfs dependencies (which is not trivial).
I've noticed that importer depends for no particular good reason on gs-wms
(just to run a test against wms getCapabilities, but the same could be done
listing directly from the catalog), there might be some minor build times
improvement in there, but right now I have no time... I might have a quick
look next week :-)

Cheers
Andrea


On Sat, Oct 14, 2017 at 1:49 AM, Jody Garnett 
wrote:

> My feedback was not a -1, given that you have a solid benefit from the
> modular architecture (secure env and micro deployment) I am firmly behind
> breaking up rest-config into modules :)
>
> --
> Jody Garnett
>
> On 13 October 2017 at 13:35, Andrea Aime 
> wrote:
>
>>
>>
>> Il 13 ott 2017 10:13 PM, "Jody Garnett"  ha
>> scritto:
>>
>> Thinking about splitting up rest-config 
>>
>> We ship the rest api out of the box currently I have not heard of any
>> deployments where people have used our modular system make a distribution
>> that does not include the rest api?
>>
>>
>> I do that regularly for customers, but those are custom builds for high
>> security environments: no ui, no rest, no way to change the fixed
>> configuration.
>>
>> In other cases I have setups that a more micro-service oriented where the
>> war supposedly contains a single service.
>>
>> Finally, the UI is already split on a per service modules, why not have
>> REST align?
>>
>>
>> The thing I like most about splitting up rest-config is it would allow
>> the application to be more modular. If the rest-api is always present is it
>> worth adding the "rest-config" into the individual gs-wfs and gs-wms
>> modules?
>>
>>
>> Besides the above feedback, even if rest config was always in, the per
>> service modules would be in the core packaging anyways of course. So, where
>> would be the harm, and how would it offset the benefit of a faster build to
>> the point that we should abandon the idea? (your feedback sounds like a -1)
>>
>> Regards
>> Andrea
>>
>>
>


-- 

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!

Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-13 Thread Jody Garnett 
My feedback was not a -1, given that you have a solid benefit from the
modular architecture (secure env and micro deployment) I am firmly behind
breaking up rest-config into modules :)

--
Jody Garnett

On 13 October 2017 at 13:35, Andrea Aime 
wrote:

>
>
> Il 13 ott 2017 10:13 PM, "Jody Garnett"  ha
> scritto:
>
> Thinking about splitting up rest-config 
>
> We ship the rest api out of the box currently I have not heard of any
> deployments where people have used our modular system make a distribution
> that does not include the rest api?
>
>
> I do that regularly for customers, but those are custom builds for high
> security environments: no ui, no rest, no way to change the fixed
> configuration.
>
> In other cases I have setups that a more micro-service oriented where the
> war supposedly contains a single service.
>
> Finally, the UI is already split on a per service modules, why not have
> REST align?
>
>
> The thing I like most about splitting up rest-config is it would allow the
> application to be more modular. If the rest-api is always present is it
> worth adding the "rest-config" into the individual gs-wfs and gs-wms
> modules?
>
>
> Besides the above feedback, even if rest config was always in, the per
> service modules would be in the core packaging anyways of course. So, where
> would be the harm, and how would it offset the benefit of a faster build to
> the point that we should abandon the idea? (your feedback sounds like a -1)
>
> Regards
> Andrea
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-13 Thread Andrea Aime 
Il 13 ott 2017 10:13 PM, "Jody Garnett"  ha scritto:

Thinking about splitting up rest-config 

We ship the rest api out of the box currently I have not heard of any
deployments where people have used our modular system make a distribution
that does not include the rest api?


I do that regularly for customers, but those are custom builds for high
security environments: no ui, no rest, no way to change the fixed
configuration.

In other cases I have setups that a more micro-service oriented where the
war supposedly contains a single service.

Finally, the UI is already split on a per service modules, why not have
REST align?


The thing I like most about splitting up rest-config is it would allow the
application to be more modular. If the rest-api is always present is it
worth adding the "rest-config" into the individual gs-wfs and gs-wms
modules?


Besides the above feedback, even if rest config was always in, the per
service modules would be in the core packaging anyways of course. So, where
would be the harm, and how would it offset the benefit of a faster build to
the point that we should abandon the idea? (your feedback sounds like a -1)

Regards
Andrea
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-13 Thread Jody Garnett 
Thinking about splitting up rest-config 

We ship the rest api out of the box currently I have not heard of any
deployments where people have used our modular system make a distribution
that does not include the rest api?

The thing I like most about splitting up rest-config is it would allow the
application to be more modular. If the rest-api is always present is it
worth adding the "rest-config" into the individual gs-wfs and gs-wms
modules?


--
Jody Garnett

On 11 October 2017 at 09:36, Andrea Aime 
wrote:

> Hi,
> following up the geotools-devel thread on speeding up the build, I've made
> some checks on
> GeoServer as well.
>
> 
> Please let's not start a discussion about tests being integration and thus
> slow, we all know that,
> but I'm trying to discuss something that can be done with limited
> resources here,
> not planning the next 10 years of development or selling an idea for the
> next code sprint :-)
> (it's a fair conversation topic, if you want to start it please just do so
> in a separate mail thread).
> 
>
> Compared to GeoTools the build takes longer because, among other reasons,
> it parallelizes a lot less.
> The "chain of death" keeping the build in low parallel mode is, roughly:
> main->wfs->wms->restconfig->app-schema-test
>
> The main module takes here 1:20, and everything else depends on it. A
> large chunk of that time is
> spent running security related tests.
> I'm only semi serious here, but as far as I can see, it could be
> beneficial to move the securty
> tests to their own module (e.g., security/core) while leaving the security
> classes in main.
> I'm just semi-serious because I realize splitting apart the classes and
> their tests is not that great,
> but at the same time, it could easily shave off 30-60 seconds off the
> critical path.
>
> The WMS dependency onto WFS is hard to un-tangle, I've tried already some
> years ago and
> eventually gave up, it would require its own mini-sprint imho... so off
> the board for this discussion.
>
> Restconfig is where things get interesting and actually doable in a
> relatively short time.
> Restconfig is a slow module to build, and has to wait onto all core OGC
> services to be built, but it
> does not have to be that way: I propose we split the service support part
> into a small per service
> module, e.g.:
>
>- gs-restconfig-wfs
>- gs-restconfig-wms
>- gs-restconfig-wcs
>- gs-restconfig-wps (missing today, no way to configure WPS from REST)
>- gs-restconfig-csw (missing today, no way to configure CSW from REST)
>
> Basically gs-restconfig would contain the core abstract service controller
> class, and the "one class modules"
> would add the templates, extra code, and tests (side effect, we'd make the
> service rest configuration
> actually pluggable).
> There are also 4-5 a few tests hitting the OGC services as part of the
> tests, those could also be moved
> to the service specific restconfig module.
> This way the main gs-restconfig module could start building as soon as
> gs-main has completed.
>
> The other module that should be split imho is the app-schema-test one, in
> the current build, and
> on my current machine, it runs alone for 55 seconds after any other module
> has terminated building.
> The reason being that it depends on too many things, wfs, wms, restconfig.
> I believe the module mostly does WFS tests with a few WMS and I believe
> just one REST test, so can
> we split it into three parts that can start running as soon as their
> dependencies are made available?
>
>- app-schema-wfs-test
>- app-schema-wms-test
>- app-schema-rest-test
>
> Cheers
> Andrea
>
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313 <+39%200584%20962313>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
> mob: +39  339 8844549 <+39%20339%20884%204549>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the

Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-13 Thread bradh 
I certainly have no objections to things being faster.

 

My only concern was on the geotools side, if there was an API change. I don’t 
see that as a consideration on geoserver though (application vs library).

 

I’m not really able to help with it at the moment though, although I may be 
able to apply some test effort.

 

Brad

 

From: Andrea Aime [mailto:andrea.a...@geo-solutions.it] 
Sent: Saturday, 14 October 2017 2:30 AM
To: Geoserver-devel <geoserver-devel@lists.sourceforge.net>
Subject: Re: [Geoserver-devel] Speeding up the GeoServer build by splitting 
some modules

 

Hi,

as with the thread on geotools-devel, there seems to be a distinct lack of 
enthusiams

for the topic.

Can at least get a sense if there are people opposed to the changes I'm 
proposing?

I'd like to know before starting to split modules apart :-)

 

Cheers

Andreda

 

 

On Wed, Oct 11, 2017 at 6:36 PM, Andrea Aime <andrea.a...@geo-solutions.it 
<mailto:andrea.a...@geo-solutions.it> > wrote:

Hi,

following up the geotools-devel thread on speeding up the build, I've made some 
checks on 

GeoServer as well.

 



Please let's not start a discussion about tests being integration and thus 
slow, we all know that,

but I'm trying to discuss something that can be done with limited resources 
here,

not planning the next 10 years of development or selling an idea for the next 
code sprint :-)

(it's a fair conversation topic, if you want to start it please just do so in a 
separate mail thread).



 

Compared to GeoTools the build takes longer because, among other reasons, it 
parallelizes a lot less.

The "chain of death" keeping the build in low parallel mode is, roughly:

main->wfs->wms->restconfig->app-schema-test

 

The main module takes here 1:20, and everything else depends on it. A large 
chunk of that time is

spent running security related tests.

I'm only semi serious here, but as far as I can see, it could be beneficial to 
move the securty

tests to their own module (e.g., security/core) while leaving the security 
classes in main.

I'm just semi-serious because I realize splitting apart the classes and their 
tests is not that great,

but at the same time, it could easily shave off 30-60 seconds off the critical 
path.

 

The WMS dependency onto WFS is hard to un-tangle, I've tried already some years 
ago and

eventually gave up, it would require its own mini-sprint imho... so off the 
board for this discussion.

 

Restconfig is where things get interesting and actually doable in a relatively 
short time.

Restconfig is a slow module to build, and has to wait onto all core OGC 
services to be built, but it

does not have to be that way: I propose we split the service support part into 
a small per service

module, e.g.:

*   gs-restconfig-wfs
*   gs-restconfig-wms
*   gs-restconfig-wcs
*   gs-restconfig-wps (missing today, no way to configure WPS from REST)
*   gs-restconfig-csw (missing today, no way to configure CSW from REST)

Basically gs-restconfig would contain the core abstract service controller 
class, and the "one class modules"

would add the templates, extra code, and tests (side effect, we'd make the 
service rest configuration

actually pluggable). 

There are also 4-5 a few tests hitting the OGC services as part of the tests, 
those could also be moved

to the service specific restconfig module.

This way the main gs-restconfig module could start building as soon as gs-main 
has completed.

 

The other module that should be split imho is the app-schema-test one, in the 
current build, and 

on my current machine, it runs alone for 55 seconds after any other module has 
terminated building.

The reason being that it depends on too many things, wfs, wms, restconfig. 

I believe the module mostly does WFS tests with a few WMS and I believe just 
one REST test, so can

we split it into three parts that can start running as soon as their 
dependencies are made available?

*   app-schema-wfs-test
*   app-schema-wms-test
*   app-schema-rest-test

Cheers

Andrea

 

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V 
for more information.
==

Ing. Andrea Aime 
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313 <tel:+39%200584%20962313> 
fax: +39 0584 1660272 <tel:+39%200584%20166%200272> 
mob: +39  <tel:+39%20339%20884%204549>  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i 
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo 
è consentito esclusivamente al destinatario del messaggio, per le finalità 
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne 
il destinatario, Vi preghiamo cortesemente di darcene noti

Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-13 Thread Justin Deoliveira 
I am obviously not in a place to approve / disapprove but I’ll add that
your breakdown makes a lot of sense to me.

On Fri, Oct 13, 2017 at 9:30 AM Andrea Aime 
wrote:

> Hi,
> as with the thread on geotools-devel, there seems to be a distinct lack of
> enthusiams
> for the topic.
> Can at least get a sense if there are people opposed to the changes I'm
> proposing?
> I'd like to know before starting to split modules apart :-)
>
> Cheers
> Andreda
>
>
> On Wed, Oct 11, 2017 at 6:36 PM, Andrea Aime  > wrote:
>
>> Hi,
>> following up the geotools-devel thread on speeding up the build, I've
>> made some checks on
>> GeoServer as well.
>>
>> 
>> Please let's not start a discussion about tests being integration and
>> thus slow, we all know that,
>> but I'm trying to discuss something that can be done with limited
>> resources here,
>> not planning the next 10 years of development or selling an idea for the
>> next code sprint :-)
>> (it's a fair conversation topic, if you want to start it please just do
>> so in a separate mail thread).
>> 
>>
>> Compared to GeoTools the build takes longer because, among other reasons,
>> it parallelizes a lot less.
>> The "chain of death" keeping the build in low parallel mode is, roughly:
>> main->wfs->wms->restconfig->app-schema-test
>>
>> The main module takes here 1:20, and everything else depends on it. A
>> large chunk of that time is
>> spent running security related tests.
>> I'm only semi serious here, but as far as I can see, it could be
>> beneficial to move the securty
>> tests to their own module (e.g., security/core) while leaving the
>> security classes in main.
>> I'm just semi-serious because I realize splitting apart the classes and
>> their tests is not that great,
>> but at the same time, it could easily shave off 30-60 seconds off the
>> critical path.
>>
>> The WMS dependency onto WFS is hard to un-tangle, I've tried already some
>> years ago and
>> eventually gave up, it would require its own mini-sprint imho... so off
>> the board for this discussion.
>>
>> Restconfig is where things get interesting and actually doable in a
>> relatively short time.
>> Restconfig is a slow module to build, and has to wait onto all core OGC
>> services to be built, but it
>> does not have to be that way: I propose we split the service support part
>> into a small per service
>> module, e.g.:
>>
>>- gs-restconfig-wfs
>>- gs-restconfig-wms
>>- gs-restconfig-wcs
>>- gs-restconfig-wps (missing today, no way to configure WPS from REST)
>>- gs-restconfig-csw (missing today, no way to configure CSW from REST)
>>
>> Basically gs-restconfig would contain the core abstract service
>> controller class, and the "one class modules"
>> would add the templates, extra code, and tests (side effect, we'd make
>> the service rest configuration
>> actually pluggable).
>> There are also 4-5 a few tests hitting the OGC services as part of the
>> tests, those could also be moved
>> to the service specific restconfig module.
>> This way the main gs-restconfig module could start building as soon as
>> gs-main has completed.
>>
>> The other module that should be split imho is the app-schema-test one, in
>> the current build, and
>> on my current machine, it runs alone for 55 seconds after any other
>> module has terminated building.
>> The reason being that it depends on too many things, wfs, wms,
>> restconfig.
>> I believe the module mostly does WFS tests with a few WMS and I believe
>> just one REST test, so can
>> we split it into three parts that can start running as soon as their
>> dependencies are made available?
>>
>>- app-schema-wfs-test
>>- app-schema-wms-test
>>- app-schema-rest-test
>>
>> Cheers
>> Andrea
>>
>> ==
>> GeoServer Professional Services from the experts! Visit
>> http://goo.gl/it488V for more information.
>> ==
>>
>> Ing. Andrea Aime
>> @geowolf
>> Technical Lead
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> phone: +39 0584 962313 <+39%200584%20962313>
>> fax: +39 0584 1660272 <+39%200584%20166%200272>
>> mob: +39  339 8844549 <+39%20339%20884%204549>
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>>
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>> principi

Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-13 Thread Andrea Aime 
Hi,
as with the thread on geotools-devel, there seems to be a distinct lack of
enthusiams
for the topic.
Can at least get a sense if there are people opposed to the changes I'm
proposing?
I'd like to know before starting to split modules apart :-)

Cheers
Andreda


On Wed, Oct 11, 2017 at 6:36 PM, Andrea Aime 
wrote:

> Hi,
> following up the geotools-devel thread on speeding up the build, I've made
> some checks on
> GeoServer as well.
>
> 
> Please let's not start a discussion about tests being integration and thus
> slow, we all know that,
> but I'm trying to discuss something that can be done with limited
> resources here,
> not planning the next 10 years of development or selling an idea for the
> next code sprint :-)
> (it's a fair conversation topic, if you want to start it please just do so
> in a separate mail thread).
> 
>
> Compared to GeoTools the build takes longer because, among other reasons,
> it parallelizes a lot less.
> The "chain of death" keeping the build in low parallel mode is, roughly:
> main->wfs->wms->restconfig->app-schema-test
>
> The main module takes here 1:20, and everything else depends on it. A
> large chunk of that time is
> spent running security related tests.
> I'm only semi serious here, but as far as I can see, it could be
> beneficial to move the securty
> tests to their own module (e.g., security/core) while leaving the security
> classes in main.
> I'm just semi-serious because I realize splitting apart the classes and
> their tests is not that great,
> but at the same time, it could easily shave off 30-60 seconds off the
> critical path.
>
> The WMS dependency onto WFS is hard to un-tangle, I've tried already some
> years ago and
> eventually gave up, it would require its own mini-sprint imho... so off
> the board for this discussion.
>
> Restconfig is where things get interesting and actually doable in a
> relatively short time.
> Restconfig is a slow module to build, and has to wait onto all core OGC
> services to be built, but it
> does not have to be that way: I propose we split the service support part
> into a small per service
> module, e.g.:
>
>- gs-restconfig-wfs
>- gs-restconfig-wms
>- gs-restconfig-wcs
>- gs-restconfig-wps (missing today, no way to configure WPS from REST)
>- gs-restconfig-csw (missing today, no way to configure CSW from REST)
>
> Basically gs-restconfig would contain the core abstract service controller
> class, and the "one class modules"
> would add the templates, extra code, and tests (side effect, we'd make the
> service rest configuration
> actually pluggable).
> There are also 4-5 a few tests hitting the OGC services as part of the
> tests, those could also be moved
> to the service specific restconfig module.
> This way the main gs-restconfig module could start building as soon as
> gs-main has completed.
>
> The other module that should be split imho is the app-schema-test one, in
> the current build, and
> on my current machine, it runs alone for 55 seconds after any other module
> has terminated building.
> The reason being that it depends on too many things, wfs, wms, restconfig.
> I believe the module mostly does WFS tests with a few WMS and I believe
> just one REST test, so can
> we split it into three parts that can start running as soon as their
> dependencies are made available?
>
>- app-schema-wfs-test
>- app-schema-wms-test
>- app-schema-rest-test
>
> Cheers
> Andrea
>
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313 <+39%200584%20962313>
> fax: +39 0584 1660272 <+39%200584%20166%200272>
> mob: +39  339 8844549 <+39%20339%20884%204549>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any

[Geoserver-devel] Speeding up the GeoServer build by splitting some modules

2017-10-11 Thread Andrea Aime 
Hi,
following up the geotools-devel thread on speeding up the build, I've made
some checks on
GeoServer as well.


Please let's not start a discussion about tests being integration and thus
slow, we all know that,
but I'm trying to discuss something that can be done with limited resources
here,
not planning the next 10 years of development or selling an idea for the
next code sprint :-)
(it's a fair conversation topic, if you want to start it please just do so
in a separate mail thread).


Compared to GeoTools the build takes longer because, among other reasons,
it parallelizes a lot less.
The "chain of death" keeping the build in low parallel mode is, roughly:
main->wfs->wms->restconfig->app-schema-test

The main module takes here 1:20, and everything else depends on it. A large
chunk of that time is
spent running security related tests.
I'm only semi serious here, but as far as I can see, it could be beneficial
to move the securty
tests to their own module (e.g., security/core) while leaving the security
classes in main.
I'm just semi-serious because I realize splitting apart the classes and
their tests is not that great,
but at the same time, it could easily shave off 30-60 seconds off the
critical path.

The WMS dependency onto WFS is hard to un-tangle, I've tried already some
years ago and
eventually gave up, it would require its own mini-sprint imho... so off the
board for this discussion.

Restconfig is where things get interesting and actually doable in a
relatively short time.
Restconfig is a slow module to build, and has to wait onto all core OGC
services to be built, but it
does not have to be that way: I propose we split the service support part
into a small per service
module, e.g.:

   - gs-restconfig-wfs
   - gs-restconfig-wms
   - gs-restconfig-wcs
   - gs-restconfig-wps (missing today, no way to configure WPS from REST)
   - gs-restconfig-csw (missing today, no way to configure CSW from REST)

Basically gs-restconfig would contain the core abstract service controller
class, and the "one class modules"
would add the templates, extra code, and tests (side effect, we'd make the
service rest configuration
actually pluggable).
There are also 4-5 a few tests hitting the OGC services as part of the
tests, those could also be moved
to the service specific restconfig module.
This way the main gs-restconfig module could start building as soon as
gs-main has completed.

The other module that should be split imho is the app-schema-test one, in
the current build, and
on my current machine, it runs alone for 55 seconds after any other module
has terminated building.
The reason being that it depends on too many things, wfs, wms, restconfig.
I believe the module mostly does WFS tests with a few WMS and I believe
just one REST test, so can
we split it into three parts that can start running as soon as their
dependencies are made available?

   - app-schema-wfs-test
   - app-schema-wms-test
   - app-schema-rest-test

Cheers
Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail