Hello.
As mentioned before, there are two problems with the WMS GetCapabilities
operation. The first problem is that one of the wrappers
(OracleDatastoreWrapper or PostgisDatastoreWrapper) between GeoServer
and the datastore is not passing down the visitor (MinVisitor,
MaxVisitor, UniqueVisitor, etc.) and thus broke the optimization
selecting granules from the datastore. The problem has been solved by
adding a "accepts" method to the TransformFeatureCollection class which
allows to pass down the visitor and use optimized queries (see
https://github.com/geotools/geotools/pull/2009). The second problem why
GetCapabilities query works not enough fast is still not solved and
needs to be discussed.
Details:
Our performance problem are caused by the queries to select the values
of the dimensions. We have compared queries for the dimensions which
have the different values of the properties "Presentation": "List" and
"Interval and resolution". It was realized that the queries selecting
the values of the dimension both presentations are the same and have the
following view:
PostGIS (with WrapStore=false): SELECT distinct("<dim_name>")
FROM "<table_name>"
Oracle: SELECT distinct(<dim_name>) FROM <TABLE_NAME>
PostGIS (with WrapStore=true): SELECT distinct("<dim_name>")
FROM "<table_name>"
But if the "Interval and resolution" presentation of the dimension is
selected then only the min/max values are really needed, which could be
calculated much faster in the database.
Solutions:
The main idea is to inform GeoTools level modules about what value of
the dimension presentation is selected and to change the visitors on the
fly if the "Interval and resolution" dimension presentation was
selected. The following changes of the gs-main module of the Geoserver
and the gt-metadata and gt-imagemosaic modules of the GeoTools were
proposed.
Information about dimension configuration are saving in the object of
the CoverageInfo class. Object controls and invokes visitors to extract
dimensions values instantiated RasterManager class. The CoverageInfo
object and RasterManager object have the common namespace in the
CoverDimensionCustomizerReader.java in the method wrap. The value of the
dimension presentation can be sent from the GeoServer to the GeoTools
using new RenderingHints property in the Hints object. The following is
the proposed changes:
- in the CoverDimensionCustomizerReader.java method (GeoServer)
following code should be appended:
public static GridCoverageReader wrap(GridCoverage2DReader delegate,
String coverageName, CoverageInfo info) {
GridCoverage2DReader reader = delegate;
if (coverageName != null) {
reader = SingleGridCoverage2DReader.wrap(delegate,
coverageName);
}
/** Check if dimension has presentation List **/
if(info != null && delegate instanceof ImageMosaicReader) {
RasterManager manager = ((ImageMosaicReader)
delegate).getRasterManager(info.getNativeCoverageName());
if(manager != null) {
Map<String, Boolean> dimPresentationData = new HashMap<>();
manager.getHints().add(new
RenderingHints(Hints.DIMENSIONS_PRESENTATIONS_INFO,dimPresentationData));
for (Map.Entry<String, Serializable> e :
info.getMetadata().entrySet()) {
String key = e.getKey();
Object value = e.getValue();
if
(key.equals(org.geoserver.catalog.ResourceInfo.TIME)) {
DimensionInfo timeInfo =
Converters.convert(value, DimensionInfo.class);
dimPresentationData.put(key,
DimensionPresentation.LIST ==
timeInfo.getPresentation());
} else if
(key.equals(org.geoserver.catalog.ResourceInfo.ELEVATION)) {
DimensionInfo elevInfo
=Converters.convert(value, DimensionInfo.class);
dimPresentationData.put(key,
DimensionPresentation.LIST ==
elevInfo.getPresentation());
} else if (value instanceof DimensionInfo) {
DimensionInfo dimInfo = (DimensionInfo) value;
if (dimInfo.isEnabled()) {
if
(key.startsWith(org.geoserver.catalog.ResourceInfo.CUSTOM_DIMENSION_PREFIX))
{
String dimensionName =
key.substring(org.geoserver
.catalog
.ResourceInfo
.CUSTOM_DIMENSION_PREFIX
.length());
dimPresentationData.put(dimensionName.toLowerCase(),
DimensionPresentation.LIST ==
dimInfo.getPresentation());
}
}
}
}
}
}
/************************************************/
if (reader instanceof StructuredGridCoverage2DReader) {
return new
CoverageDimensionCustomizerStructuredReader((StructuredGridCoverage2DReader)
reader, coverageName, info);
} else {
return new CoverageDimensionCustomizerReader(reader,
coverageName, info);
}
}
- in the Hints.java (GeoTools) following code should be appended:
public static final ClassKey DIMENSIONS_PRESENTATIONS_INFO = new
ClassKey("java.util.Map");
The following changes of the code allow to control value of selected
presentation and allow to change the visitors to select only the minimum
and the maximum values if the "Interval and resolution" dimension
presentation was selected.
- in the RasterManager.java (GeoTools) following code should be appended:
Set extractDomain(final String attribute) throws IOException {
Query query = new Query(typeName);
query.setPropertyNames(Arrays.asList(attribute));
final UniqueVisitor visitor = new UniqueVisitor(attribute);
/*** control the presentation of the dimansion ***/
Hints rmHints = getHints();
if (rmHints.containsKey(Hints.DIMENSIONS_PRESENTATIONS_INFO)) {
Map<String, Boolean> dimPresentationParam =
(Map<String, Boolean>)
rmHints.get(Hints.DIMENSIONS_PRESENTATIONS_INFO);
if (dimPresentationParam != null) {
Boolean isListPresentation =
dimPresentationParam.get(attribute);
if (!isListPresentation) {
final FeatureCalc[] visitors = {
new MinVisitor(attribute), new
MaxVisitor(attribute)
};
Set values = new HashSet();
for (FeatureCalc extremVisitor : visitors) {
granuleCatalog.computeAggregateFunction(query, extremVisitor);
values.add(extremVisitor.getResult().getValue());
}
return values;
}
}
}
/*******/
granuleCatalog.computeAggregateFunction(query, visitor);
return visitor.getUnique();
}
We would like to get some feedback on will these changes correct or are
there another parts of GeoServer/GeoTools where GeoServer can inform
GeoTools about selected presentation and GeoTools can change the
visitors to obtain the "Interval and resolution" values of the dimension?
Best regards,
Ivan
Dňa 09.08.2018 o 18:52 Andrea Aime napísal(a):
Ah,
by looking at it, it would seems the store wrapper in the mosaic
module is not correctly
delegating down the visitor and thus breaking database optimizations.
Yep, pull requests are quite welcomed, please pay attention to the
rules to contribute,
detailed here:
https://github.com/geotools/geotools/blob/master/CONTRIBUTING.md
In particular, note code formatting (just running maven on the command
line will reformat the code as expected), presence of tests, and
contribution agreement
Cheers
Andrea
On Thu, Aug 9, 2018 at 6:30 PM György Tomcsányi
<gyorgy.tomcsa...@microstep-mis.com
<mailto:gyorgy.tomcsa...@microstep-mis.com>> wrote:
Hello all,
we are using GeoServer to display a large number of GeoTIFFs using
ImageMosaic data stores. The data has several dimensions (time and
custom). We are adding new data to these stores periodically and the
goal is to be able to display years of data (millions of
granules). We
have encountered performance problems with the WMS GetCapabilities
operation. It is most noticeable when using Oracle database (or
PostGIS
with parameter WrapStore=true). My colleague Ivan (in CC) implemented
changes which improved the performance for our use case
significantly.
We would like to publish these changes and hopefully merge them
into the
official repo.
Details:
our performance problems are caused by the queries for values and
defaults for dimensions. The currently used queries vary depending on
the selected presentation for the dimension:
1. query for values:
"List":
* PostGIS (with WrapStore=false): SELECT distinct("<dim_name>")
FROM "<table_name>"
* Oracle: SELECT FID, <DIM_NAME> FROM <table_name>
* PostGIS (with WrapStore=true): SELECT "fid","<dim_name>" FROM
"<table_name>"
First version is clearly the fastest, because it loads only
distinct values.
"Interval and resolution":
The queries are the same as for "List", but in this case we only
really need the min/max values, which could be calculated much
faster in
the database.
2. query for defaults (when using smallest/biggest domain value):
* PostGIS (with WrapStore=false): SELECT <min | max>
("<dim_name>")
FROM "<table_name>"
* Oracle: SELECT FID, <DIM_NAME> FROM <table_name>
* PostGIS (with WrapStore=true): SELECT "fid","<dim_name>" FROM
"<table_name>"
Again the first PostGIS variant is clearly the fastest. Others
load
all the values to the application where the min/max values are
calculated.
Our proposed solution creates optimized queries depending on which
presentation and default setting is used for the dimension. The
needed
additional information from the settings is sent to the lower level
functions using new RenderingHints. The goal of the optimized
queries is
to load only the needed values from the database (usually what the
current PostGIS query does). Unit tests were also added.
We would like to get some feedback on this. Can I submit merge
requests
(to both GeoTools and GeoServer) to review the code?
We are looking forward to any comments.
best regards,
Gyorgy Tomcsanyi
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel
--
Regards, Andrea Aime == GeoServer Professional Services from the
experts! Visit http://goo.gl/it488V for more information. == Ing.
Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di
Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39
0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it
http://twitter.com/geosolutions_it
------------------------------------------------------- /Con
riferimento alla normativa sul trattamento dei dati personali (Reg. UE
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si
precisa che ogni circostanza inerente alla presente email (il suo
contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è
riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il
messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra
operazione è illecita. Le sarei comunque grato se potesse darmene
notizia. This email is intended only for the person or entity to which
it is addressed and may contain information that is privileged,
confidential or otherwise protected from disclosure. We remind that -
as provided by European Regulation 2016/679 “GDPR” - copying,
dissemination or use of this e-mail or the information herein by
anyone other than the intended recipient is prohibited. If you have
received this email by mistake, please notify us immediately by
telephone or e-mail./
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
