Hi Ivan,
I believe you're going with a custom path where a general one could be
used, and the changes can be limited to GeoServer.
In the case of vector data I believe only min and max values are extracted
even in the case of min/max/resolution, the code path
starts here:
https://github.com/geoserver/geoserver/blob/6e9e25c0c7cdda9ada9f33f8255130d3afc76801/src/wms/src/main/java/org/geoserver/wms/capabilities/DimensionHelper.java#L601
For the raster case the code is dumber and indeed extracts the whole
domain, here:
https://github.com/geoserver/geoserver/blob/6e9e25c0c7cdda9ada9f33f8255130d3afc76801/src/wms/src/main/java/org/geoserver/wms/capabilities/DimensionHelper.java#L291
However, the RasterdimensionAccessor also provides getMinTime/getMaxTime
methods (and same for elevation) that can be used to
mimic the vector path behavior. Internally those should be optimized with
visits (but please double check that)
Cheers
Andrea
On Thu, Aug 23, 2018 at 10:21 AM Ivan Martynovskyi <
ivan.martynovs...@microstep-mis.com> wrote:
> Hello.
> As mentioned before, there are two problems with the WMS GetCapabilities
> operation. The first problem is that one of the wrappers
> (OracleDatastoreWrapper or PostgisDatastoreWrapper) between GeoServer and
> the datastore is not passing down the visitor (MinVisitor, MaxVisitor,
> UniqueVisitor, etc.) and thus broke the optimization selecting granules
> from the datastore. The problem has been solved by adding a "accepts"
> method to the TransformFeatureCollection class which allows to pass down
> the visitor and use optimized queries (see
> https://github.com/geotools/geotools/pull/2009). The second problem why
> GetCapabilities query works not enough fast is still not solved and needs
> to be discussed.
>
> Details:
> Our performance problem are caused by the queries to select the values of
> the dimensions. We have compared queries for the dimensions which have the
> different values of the properties "Presentation": "List" and "Interval and
> resolution". It was realized that the queries selecting the values of the
> dimension both presentations are the same and have the following view:
> PostGIS (with WrapStore=false): SELECT distinct("<dim_name>") FROM
> "<table_name>"
> Oracle: SELECT distinct(<dim_name>) FROM <TABLE_NAME>
> PostGIS (with WrapStore=true): SELECT distinct("<dim_name>") FROM
> "<table_name>"
>
> But if the "Interval and resolution" presentation of the dimension is
> selected then only the min/max values are really needed, which could be
> calculated much faster in the database.
>
> Solutions:
> The main idea is to inform GeoTools level modules about what value of the
> dimension presentation is selected and to change the visitors on the fly if
> the "Interval and resolution" dimension presentation was selected. The
> following changes of the gs-main module of the Geoserver and the
> gt-metadata and gt-imagemosaic modules of the GeoTools were proposed.
>
> Information about dimension configuration are saving in the object of the
> CoverageInfo class. Object controls and invokes visitors to extract
> dimensions values instantiated RasterManager class. The CoverageInfo object
> and RasterManager object have the common namespace in the
> CoverDimensionCustomizerReader.java in the method wrap. The value of the
> dimension presentation can be sent from the GeoServer to the GeoTools using
> new RenderingHints property in the Hints object. The following is the
> proposed changes:
> - in the CoverDimensionCustomizerReader.java method (GeoServer) following
> code should be appended:
> public static GridCoverageReader wrap(GridCoverage2DReader delegate,
> String coverageName, CoverageInfo info) {
> GridCoverage2DReader reader = delegate;
> if (coverageName != null) {
> reader = SingleGridCoverage2DReader.wrap(delegate,
> coverageName);
> }
> /** Check if dimension has presentation List **/
> if(info != null && delegate instanceof ImageMosaicReader) {
> RasterManager manager = ((ImageMosaicReader)
> delegate).getRasterManager(info.getNativeCoverageName());
> if(manager != null) {
> Map<String, Boolean> dimPresentationData = new HashMap<>();
> manager.getHints().add(new
> RenderingHints(Hints.DIMENSIONS_PRESENTATIONS_INFO,dimPresentationData));
> for (Map.Entry<String, Serializable> e :
> info.getMetadata().entrySet()) {
> String key = e.getKey();
> Object value = e.getValue();
> if
> (key.equals(org.geoserver.catalog.ResourceInfo.TIME)) {
> DimensionInfo timeInfo = Converters.convert(value,
> DimensionInfo.class);
> dimPresentationData.put(key,
> DimensionPresentation.LIST ==
> timeInfo.getPresentation());
> } else if
> (key.equals(org.geoserver.catalog.ResourceInfo.ELEVATION)) {
> DimensionInfo elevInfo =Converters.convert(value,
> DimensionInfo.class);
> dimPresentationData.put(key,
> DimensionPresentation.LIST ==
> elevInfo.getPresentation());
> } else if (value instanceof DimensionInfo) {
> DimensionInfo dimInfo = (DimensionInfo) value;
> if (dimInfo.isEnabled()) {
> if
> (key.startsWith(org.geoserver.catalog.ResourceInfo.CUSTOM_DIMENSION_PREFIX))
> {
> String dimensionName =
> key.substring(org.geoserver
> .catalog
> .ResourceInfo
> .CUSTOM_DIMENSION_PREFIX
> .length());
>
> dimPresentationData.put(dimensionName.toLowerCase(),
> DimensionPresentation.LIST ==
> dimInfo.getPresentation());
> }
> }
> }
> }
> }
> }
> /************************************************/
> if (reader instanceof StructuredGridCoverage2DReader) {
> return new
> CoverageDimensionCustomizerStructuredReader((StructuredGridCoverage2DReader)
> reader, coverageName, info);
> } else {
> return new CoverageDimensionCustomizerReader(reader,
> coverageName, info);
> }
> }
>
> - in the Hints.java (GeoTools) following code should be appended:
>
> public static final ClassKey DIMENSIONS_PRESENTATIONS_INFO = new
> ClassKey("java.util.Map");
>
> The following changes of the code allow to control value of selected
> presentation and allow to change the visitors to select only the minimum
> and the maximum values if the "Interval and resolution" dimension
> presentation was selected.
> - in the RasterManager.java (GeoTools) following code should be appended:
>
> Set extractDomain(final String attribute) throws IOException {
> Query query = new Query(typeName);
> query.setPropertyNames(Arrays.asList(attribute));
> final UniqueVisitor visitor = new UniqueVisitor(attribute);
> /*** control the presentation of the dimansion ***/
> Hints rmHints = getHints();
> if (rmHints.containsKey(Hints.DIMENSIONS_PRESENTATIONS_INFO)) {
> Map<String, Boolean> dimPresentationParam =
> (Map<String, Boolean>)
> rmHints.get(Hints.DIMENSIONS_PRESENTATIONS_INFO);
> if (dimPresentationParam != null) {
> Boolean isListPresentation =
> dimPresentationParam.get(attribute);
> if (!isListPresentation) {
> final FeatureCalc[] visitors = {
> new MinVisitor(attribute), new
> MaxVisitor(attribute)
> };
> Set values = new HashSet();
>
> for (FeatureCalc extremVisitor : visitors) {
> granuleCatalog.computeAggregateFunction(query,
> extremVisitor);
> values.add(extremVisitor.getResult().getValue());
> }
>
> return values;
> }
> }
> }
> /*******/
> granuleCatalog.computeAggregateFunction(query, visitor);
> return visitor.getUnique();
> }
>
> We would like to get some feedback on will these changes correct or are
> there another parts of GeoServer/GeoTools where GeoServer can inform
> GeoTools about selected presentation and GeoTools can change the visitors
> to obtain the "Interval and resolution" values of the dimension?
> Best regards,
> Ivan
>
> Dňa 09.08.2018 o 18:52 Andrea Aime napísal(a):
>
> Ah,
> by looking at it, it would seems the store wrapper in the mosaic module is
> not correctly
> delegating down the visitor and thus breaking database optimizations.
> Yep, pull requests are quite welcomed, please pay attention to the rules
> to contribute,
> detailed here:
> https://github.com/geotools/geotools/blob/master/CONTRIBUTING.md
>
> In particular, note code formatting (just running maven on the command
> line will reformat the code as expected), presence of tests, and
> contribution agreement
>
> Cheers
> Andrea
>
>
> On Thu, Aug 9, 2018 at 6:30 PM György Tomcsányi <
> gyorgy.tomcsa...@microstep-mis.com> wrote:
>
>> Hello all,
>>
>> we are using GeoServer to display a large number of GeoTIFFs using
>> ImageMosaic data stores. The data has several dimensions (time and
>> custom). We are adding new data to these stores periodically and the
>> goal is to be able to display years of data (millions of granules). We
>> have encountered performance problems with the WMS GetCapabilities
>> operation. It is most noticeable when using Oracle database (or PostGIS
>> with parameter WrapStore=true). My colleague Ivan (in CC) implemented
>> changes which improved the performance for our use case significantly.
>> We would like to publish these changes and hopefully merge them into the
>> official repo.
>>
>> Details:
>> our performance problems are caused by the queries for values and
>> defaults for dimensions. The currently used queries vary depending on
>> the selected presentation for the dimension:
>>
>> 1. query for values:
>> "List":
>> * PostGIS (with WrapStore=false): SELECT distinct("<dim_name>")
>> FROM "<table_name>"
>> * Oracle: SELECT FID, <DIM_NAME> FROM <table_name>
>> * PostGIS (with WrapStore=true): SELECT "fid","<dim_name>" FROM
>> "<table_name>"
>> First version is clearly the fastest, because it loads only
>> distinct values.
>>
>> "Interval and resolution":
>> The queries are the same as for "List", but in this case we only
>> really need the min/max values, which could be calculated much faster in
>> the database.
>>
>> 2. query for defaults (when using smallest/biggest domain value):
>> * PostGIS (with WrapStore=false): SELECT <min | max> ("<dim_name>")
>> FROM "<table_name>"
>> * Oracle: SELECT FID, <DIM_NAME> FROM <table_name>
>> * PostGIS (with WrapStore=true): SELECT "fid","<dim_name>" FROM
>> "<table_name>"
>> Again the first PostGIS variant is clearly the fastest. Others load
>> all the values to the application where the min/max values are calculated.
>>
>> Our proposed solution creates optimized queries depending on which
>> presentation and default setting is used for the dimension. The needed
>> additional information from the settings is sent to the lower level
>> functions using new RenderingHints. The goal of the optimized queries is
>> to load only the needed values from the database (usually what the
>> current PostGIS query does). Unit tests were also added.
>>
>> We would like to get some feedback on this. Can I submit merge requests
>> (to both GeoTools and GeoServer) to review the code?
>> We are looking forward to any comments.
>>
>> best regards,
>> Gyorgy Tomcsanyi
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> GeoTools-Devel mailing list
>> GeoTools-Devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>
>
>
> --
>
> Regards, Andrea Aime == GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime
> @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054
> Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339
> 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it
> ------------------------------------------------------- *Con riferimento
> alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 -
> Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni
> circostanza inerente alla presente email (il suo contenuto, gli eventuali
> allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i
> destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per
> errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
> sarei comunque grato se potesse darmene notizia. This email is intended
> only for the person or entity to which it is addressed and may contain
> information that is privileged, confidential or otherwise protected from
> disclosure. We remind that - as provided by European Regulation 2016/679
> “GDPR” - copying, dissemination or use of this e-mail or the information
> herein by anyone other than the intended recipient is prohibited. If you
> have received this email by mistake, please notify us immediately by
> telephone or e-mail.*
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _______________________________________________
> GeoTools-Devel mailing
> listGeoTools-Devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
>
--
Regards, Andrea Aime == GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime
@geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054
Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339
8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it
------------------------------------------------------- *Con riferimento
alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 -
Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo contenuto, gli eventuali
allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i
destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per
errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is intended
only for the person or entity to which it is addressed and may contain
information that is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European Regulation 2016/679
“GDPR” - copying, dissemination or use of this e-mail or the information
herein by anyone other than the intended recipient is prohibited. If you
have received this email by mistake, please notify us immediately by
telephone or e-mail.*
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
