Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Darell van der Voort]

Hello Daniele,

Thank you for the update, however I do not seem to get it to work. Please
let me know if i'm missing something. I start with the following initial
folder structure:

/geoserver/data/workspace//*.tiffs
/geoserver/data/workspace//datastore.properties
/geoserver/data/workspace//indexer.properties
/geoserver/data/workspace//1/*.tiffs
/geoserver/data/workspace//1/datastore.properties
/geoserver/data/workspace//1/indexer.properties
etc.

Note that in the initial folder structure there is no folder named '0', as
the imagepyramid only accepts folder structures as outputted by
gdal_retile. My datastore and indexer properties files look like:

*Datstore.properties:*
SPI=org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory
schema=plantation_ecu_sumifru_144
Loose\ bbox=true
Estimated\ extends=false
validate\ connections=true
Connection\ timeout=10
preparedStatements=true
jndiReferenceName=java:comp/env/jdbc/postgis

*Indexer.properties:*
Caching=false
TimeAttribute=ingestion
Schema=*the_geom:Polygon,location:String,ingestion:java.util.Date:Double
PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion)
TypeName=imagery0

Note the addition of 'TypeName' to the indexer.properties, which is unique
per folder level. Using this set-up Geoserver returns the following error:

file:///geoserver/data/testworkspace/imagery_pyramid/
java.lang.RuntimeException: Could not list layers for this store, an error
occurred retrieving them: null
at
org.geoserver.web.data.layer.NewLayerPageProvider.getItemsInternal(NewLayerPageProvider.java:158)
at
org.geoserver.web.data.layer.NewLayerPageProvider.getItems(NewLayerPageProvider.java:61)
at
org.geoserver.web.wicket.GeoServerDataProvider.fullSize(GeoServerDataProvider.java:243)
at
org.geoserver.web.wicket.GeoServerTablePanel$PagerDelegate.updateMatched(GeoServerTablePanel.java:583)
at
org.geoserver.web.wicket.GeoServerTablePanel$PagerDelegate.(GeoServerTablePanel.java:576)
at
org.geoserver.web.wicket.GeoServerTablePanel.(GeoServerTablePanel.java:176)
at
org.geoserver.web.wicket.GeoServerTablePanel.(GeoServerTablePanel.java:97)
at org.geoserver.web.data.layer.NewLayerPage$1.(NewLayerPage.java:105)
at org.geoserver.web.data.layer.NewLayerPage.(NewLayerPage.java:105)
at
org.geoserver.web.data.store.CoverageStoreNewPage.onSuccessfulSave(CoverageStoreNewPage.java:86)
at
org.geoserver.web.data.store.CoverageStoreNewPage.onSave(CoverageStoreNewPage.java:74)
at
org.geoserver.web.data.store.AbstractCoverageStorePage$1.onSubmit(AbstractCoverageStorePage.java:122)
at
org.apache.wicket.ajax.markup.html.form.AjaxSubmitLink$1.onSubmit(AjaxSubmitLink.java:111)
at
org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:215)
at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1309)
at org.apache.wicket.markup.html.form.Form.process(Form.java:976)
at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:797)
at
org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:171)
at
org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155)
at
org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:601)
at sun.reflect.GeneratedMethodAccessor427.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
at
org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248)
at
org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234)
at
org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:895)
at
org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
at
org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:265)
at
org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:222)
at
org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:293)
at
org.apache.wicket.protocol.http.WicketFilter.processRequestCycle(WicketFilter.java:261)
at
org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
at
org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at
org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:158)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
at

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Daniele Romagnoli]

Hi Darell,

On Mon, Oct 2, 2017 at 11:30 AM, Darell van der Voort 
wrote:

> Hello Daniele,
>
> Thank you for your input! I now use a schema per workspace, but creating a
> schema per imagepyramid would indeed solve the problem of duplicate table
> names. So do I understand correctly that there is no possibility to change
> the table names?
>

Nope, you can do that, instead.
You can specify the same "schema" name in all the level's
datastore.properties but you should specify a different "TypeName" property
in each indexer file. The specific ImageMosaic level will index granules in
that "TypeName" table.

Hope this helps.
Regards,
Daniele




> For example to: schema.imagepyramid0, schema.imagepyramid1 etc. In which
> schema is the unique workspace schema and imagepyramid is the name of the
> layer. Thank you!
>
> Kind regards,
> Darell
>
> On Mon, Oct 2, 2017 at 11:19 AM, Daniele Romagnoli  solutions.it> wrote:
>
>> Hi Darell,
>> as you said, the imagePyramid is basically a set of imageMosaics: an
>> ImageMosaic for each pyramid level.
>> Therefore, each level should have its own indexer and, optionally, a
>> datastore.properties if you want to catalog the granules into a DB instead
>> of default shapefile.
>>
>> When using DB instead of shapefile, you may want to use a single database
>> (as an instance, "pyramid") and store each level granule's index into a
>> different schema name. Therefore, the datastore.properties of each level
>> should define a different schema name, as an instance:
>>
>> pyramid/0/datastore.properties contains a "schema=level0" property
>> pyramid/1/datastore.properties contains a "schema=level1" property
>> pyramid/2/datastore.properties contains a "schema=level2" property
>>
>> Note that you should create the schemas (with "CREATE SCHEMA levelx" sql
>> statements) in advance on your DB, before configuring the imagePyramid
>> store in GeoServer.
>>
>> Hope this helps.
>> Regards,
>> Daniele
>>
>>
>>
>>
>>
>> On Mon, Oct 2, 2017 at 10:54 AM, Darell van der Voort <
>> darell...@gmail.com> wrote:
>>
>>> Hi Peter,
>>>
>>> Thank you for the examples, very helpful! Using a indexer.xml indeed
>>> seems like a better set-up. Have you ever tried this in combination with
>>> the imagepyramid plugin? I know that the imagepyramid are basically just
>>> imagemoasics per folder level. However as the folder structure is different
>>> it causes some problems. My folder structure looks like:
>>>
>>> /geoserver/data/workspace//0/.tiffs
>>> /geoserver/data/workspace//1/.tiffs
>>> /geoserver/data/workspace//2/.tiffs
>>> etc..
>>>
>>> Where the numbers stand for the pyramid levels which are filled with
>>> multiple tiffs, the indexer.properties and the foldername.properties. I put
>>> tiffs from different dates in the same folder and use a unique date in the
>>> filename. Should I put the indexer.xml and datastore.properties in each
>>> pyramid directory? I can try it later this week and will let you know if it
>>> works.
>>>
>>> Kind regards,
>>> Darell
>>>
>>> On Fri, Sep 29, 2017 at 11:43 AM, Peter Kovac <
>>> peter.ko...@microstep-mis.com> wrote:
>>>
 I usually configure ImageMosaic for time-series plain GeoTIFF granules
 so my advice might not be 100% accurate but I hope it helps anyway.

 Firstly, your schema seems wrong to me: you have two types specified
 for ingestion - java.util.Date and a Double after that. Perhaps another
 attribute of double type is missing or the :Double part should be deleted.

 Otherwise, I recommend using indexer.xml instead of indexer.properties
 for configuration. In your case it might look like this:

 
 

   
  ingestion<
 /attribute>
   


   
 *the_geom:Polygon,location:String,ingestion:java
 .util.Date
   


   
  your_coverage_name
  
  
 
  
   


   
  
 regex=([0-9]{14}),format=MMddHHmmss,fullPath=true
  TimestampFileNameExtractorSPI
  ingestion
   


 
 
 
 >>> value="/path/to/directory/to/be/indexed" />
 
 
 
 
 
 Usually my ImageMosaic folder structure is like this (
 /opt/geoserver_data is the GEOSERVER_DATA_DIR)
 * the indexer.xml and datastore.properties files are in a
 /opt/geoserver_data/data/
 * there is a subdirectory, usually with a date tree structure for
 storing granules from separate days in separate directories:
 /opt/geoserver_data/data//tiff//M
 M/DD/some_file_MMDDHHmss.tiff
 * in indexer.xml I have *Recursive *parameter set to
 *true ** I usually use absolute path to the tiff directory as
 *IndexingDirectories* parameter value, 

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Darell van der Voort]

Hello Daniele,

Thank you for your input! I now use a schema per workspace, but creating a
schema per imagepyramid would indeed solve the problem of duplicate table
names. So do I understand correctly that there is no possibility to change
the table names? For example to: schema.imagepyramid0, schema.imagepyramid1
etc. In which schema is the unique workspace schema and imagepyramid is the
name of the layer. Thank you!

Kind regards,
Darell

On Mon, Oct 2, 2017 at 11:19 AM, Daniele Romagnoli <
daniele.romagn...@geo-solutions.it> wrote:

> Hi Darell,
> as you said, the imagePyramid is basically a set of imageMosaics: an
> ImageMosaic for each pyramid level.
> Therefore, each level should have its own indexer and, optionally, a
> datastore.properties if you want to catalog the granules into a DB instead
> of default shapefile.
>
> When using DB instead of shapefile, you may want to use a single database
> (as an instance, "pyramid") and store each level granule's index into a
> different schema name. Therefore, the datastore.properties of each level
> should define a different schema name, as an instance:
>
> pyramid/0/datastore.properties contains a "schema=level0" property
> pyramid/1/datastore.properties contains a "schema=level1" property
> pyramid/2/datastore.properties contains a "schema=level2" property
>
> Note that you should create the schemas (with "CREATE SCHEMA levelx" sql
> statements) in advance on your DB, before configuring the imagePyramid
> store in GeoServer.
>
> Hope this helps.
> Regards,
> Daniele
>
>
>
>
>
> On Mon, Oct 2, 2017 at 10:54 AM, Darell van der Voort  > wrote:
>
>> Hi Peter,
>>
>> Thank you for the examples, very helpful! Using a indexer.xml indeed
>> seems like a better set-up. Have you ever tried this in combination with
>> the imagepyramid plugin? I know that the imagepyramid are basically just
>> imagemoasics per folder level. However as the folder structure is different
>> it causes some problems. My folder structure looks like:
>>
>> /geoserver/data/workspace//0/.tiffs
>> /geoserver/data/workspace//1/.tiffs
>> /geoserver/data/workspace//2/.tiffs
>> etc..
>>
>> Where the numbers stand for the pyramid levels which are filled with
>> multiple tiffs, the indexer.properties and the foldername.properties. I put
>> tiffs from different dates in the same folder and use a unique date in the
>> filename. Should I put the indexer.xml and datastore.properties in each
>> pyramid directory? I can try it later this week and will let you know if it
>> works.
>>
>> Kind regards,
>> Darell
>>
>> On Fri, Sep 29, 2017 at 11:43 AM, Peter Kovac <
>> peter.ko...@microstep-mis.com> wrote:
>>
>>> I usually configure ImageMosaic for time-series plain GeoTIFF granules
>>> so my advice might not be 100% accurate but I hope it helps anyway.
>>>
>>> Firstly, your schema seems wrong to me: you have two types specified for
>>> ingestion - java.util.Date and a Double after that. Perhaps another
>>> attribute of double type is missing or the :Double part should be deleted.
>>>
>>> Otherwise, I recommend using indexer.xml instead of indexer.properties
>>> for configuration. In your case it might look like this:
>>>
>>> 
>>> 
>>>
>>>   
>>>  ingestion<
>>> /attribute>
>>>   
>>>
>>>
>>>   
>>> *the_geom:Polygon,location:String,ingestion:java
>>> .util.Date
>>>   
>>>
>>>
>>>   
>>>  your_coverage_name
>>>  
>>>  
>>> 
>>>  
>>>   
>>>
>>>
>>>   
>>>  
>>> regex=([0-9]{14}),format=MMddHHmmss,fullPath=true
>>>  TimestampFileNameExtractorSPI
>>>  ingestion
>>>   
>>>
>>>
>>> 
>>> 
>>> 
>>> >> value="/path/to/directory/to/be/indexed" />
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Usually my ImageMosaic folder structure is like this (
>>> /opt/geoserver_data is the GEOSERVER_DATA_DIR)
>>> * the indexer.xml and datastore.properties files are in a
>>> /opt/geoserver_data/data/
>>> * there is a subdirectory, usually with a date tree structure for
>>> storing granules from separate days in separate directories:
>>> /opt/geoserver_data/data//tiff//M
>>> M/DD/some_file_MMDDHHmss.tiff
>>> * in indexer.xml I have *Recursive *parameter set to
>>> *true ** I usually use absolute path to the tiff directory as
>>> *IndexingDirectories* parameter value, e.g.
>>> /opt/geoserver_data/data//tiff
>>>
>>> Finally, if I encounter any problems during ImageMosaic creation I
>>> ensure to:
>>> * set GeoServer logging to VERBOSE
>>> * drop any database table created (ImageMosaic will refuse to create
>>> store if a table with the same name already exists), along with index and
>>> metadata (if using Oracle)
>>> * try it again and look into the log (usually the problem is not
>>> revealed in the last stacktrace in the log, but in some before last)
>>> * if it doesn't help or I still cannot figure out what 

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Daniele Romagnoli]

Hi Darell,
as you said, the imagePyramid is basically a set of imageMosaics: an
ImageMosaic for each pyramid level.
Therefore, each level should have its own indexer and, optionally, a
datastore.properties if you want to catalog the granules into a DB instead
of default shapefile.

When using DB instead of shapefile, you may want to use a single database
(as an instance, "pyramid") and store each level granule's index into a
different schema name. Therefore, the datastore.properties of each level
should define a different schema name, as an instance:

pyramid/0/datastore.properties contains a "schema=level0" property
pyramid/1/datastore.properties contains a "schema=level1" property
pyramid/2/datastore.properties contains a "schema=level2" property

Note that you should create the schemas (with "CREATE SCHEMA levelx" sql
statements) in advance on your DB, before configuring the imagePyramid
store in GeoServer.

Hope this helps.
Regards,
Daniele





On Mon, Oct 2, 2017 at 10:54 AM, Darell van der Voort 
wrote:

> Hi Peter,
>
> Thank you for the examples, very helpful! Using a indexer.xml indeed seems
> like a better set-up. Have you ever tried this in combination with the
> imagepyramid plugin? I know that the imagepyramid are basically just
> imagemoasics per folder level. However as the folder structure is different
> it causes some problems. My folder structure looks like:
>
> /geoserver/data/workspace//0/.tiffs
> /geoserver/data/workspace//1/.tiffs
> /geoserver/data/workspace//2/.tiffs
> etc..
>
> Where the numbers stand for the pyramid levels which are filled with
> multiple tiffs, the indexer.properties and the foldername.properties. I put
> tiffs from different dates in the same folder and use a unique date in the
> filename. Should I put the indexer.xml and datastore.properties in each
> pyramid directory? I can try it later this week and will let you know if it
> works.
>
> Kind regards,
> Darell
>
> On Fri, Sep 29, 2017 at 11:43 AM, Peter Kovac <
> peter.ko...@microstep-mis.com> wrote:
>
>> I usually configure ImageMosaic for time-series plain GeoTIFF granules so
>> my advice might not be 100% accurate but I hope it helps anyway.
>>
>> Firstly, your schema seems wrong to me: you have two types specified for
>> ingestion - java.util.Date and a Double after that. Perhaps another
>> attribute of double type is missing or the :Double part should be deleted.
>>
>> Otherwise, I recommend using indexer.xml instead of indexer.properties
>> for configuration. In your case it might look like this:
>>
>> 
>> 
>>
>>   
>>  ingestion<
>> /attribute>
>>   
>>
>>
>>   
>> *the_geom:Polygon,location:String,ingestion:java
>> .util.Date
>>   
>>
>>
>>   
>>  your_coverage_name
>>  
>>  
>> 
>>  
>>   
>>
>>
>>   
>>  
>> regex=([0-9]{14}),format=MMddHHmmss,fullPath=true
>>  TimestampFileNameExtractorSPI
>>  ingestion
>>   
>>
>>
>> 
>> 
>> 
>> > value="/path/to/directory/to/be/indexed" />
>> 
>> 
>> 
>> 
>> 
>> Usually my ImageMosaic folder structure is like this (/opt/geoserver_data
>> is the GEOSERVER_DATA_DIR)
>> * the indexer.xml and datastore.properties files are in a
>> /opt/geoserver_data/data/
>> * there is a subdirectory, usually with a date tree structure for storing
>> granules from separate days in separate directories:
>> /opt/geoserver_data/data//tiff//
>> MM/DD/some_file_MMDDHHmss.tiff
>> * in indexer.xml I have *Recursive *parameter set to
>> *true ** I usually use absolute path to the tiff directory as
>> *IndexingDirectories* parameter value, e.g.
>> /opt/geoserver_data/data//tiff
>>
>> Finally, if I encounter any problems during ImageMosaic creation I ensure
>> to:
>> * set GeoServer logging to VERBOSE
>> * drop any database table created (ImageMosaic will refuse to create
>> store if a table with the same name already exists), along with index and
>> metadata (if using Oracle)
>> * try it again and look into the log (usually the problem is not revealed
>> in the last stacktrace in the log, but in some before last)
>> * if it doesn't help or I still cannot figure out what is wrong I launch
>> Eclipse debugger and examine what EXACTLY is wrong (I used it a lot until I
>> figured out how to properly configure time parsing regexes)
>>
>> Regards,
>>
>> Peter Kovac
>>
>>
>>
>> On 29. 9. 2017 10:40, Darell van der Voort wrote:
>>
>> Hi Peter,
>>
>> This is my indexer.properties configuration:
>>
>> Caching=false
>> TimeAttribute=ingestion
>> Schema=*the_geom:Polygon,location:String,ingestion:java.util.Date:Double
>> PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion)
>>
>> And this is my 0.properties which is automatically generated for the
>> lowest zoomlevel (0):
>>
>> #-Automagically created from GeoTools-
>> #Wed Sep 27 14:50:18 UTC 

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Darell van der Voort]

Hi Peter,

Thank you for the examples, very helpful! Using a indexer.xml indeed seems
like a better set-up. Have you ever tried this in combination with the
imagepyramid plugin? I know that the imagepyramid are basically just
imagemoasics per folder level. However as the folder structure is different
it causes some problems. My folder structure looks like:

/geoserver/data/workspace//0/.tiffs
/geoserver/data/workspace//1/.tiffs
/geoserver/data/workspace//2/.tiffs
etc..

Where the numbers stand for the pyramid levels which are filled with
multiple tiffs, the indexer.properties and the foldername.properties. I put
tiffs from different dates in the same folder and use a unique date in the
filename. Should I put the indexer.xml and datastore.properties in each
pyramid directory? I can try it later this week and will let you know if it
works.

Kind regards,
Darell

On Fri, Sep 29, 2017 at 11:43 AM, Peter Kovac  wrote:

> I usually configure ImageMosaic for time-series plain GeoTIFF granules so
> my advice might not be 100% accurate but I hope it helps anyway.
>
> Firstly, your schema seems wrong to me: you have two types specified for
> ingestion - java.util.Date and a Double after that. Perhaps another
> attribute of double type is missing or the :Double part should be deleted.
>
> Otherwise, I recommend using indexer.xml instead of indexer.properties for
> configuration. In your case it might look like this:
>
> 
> 
>
>   
>  ingestion<
> /attribute>
>   
>
>
>   
> *the_geom:Polygon,location:String,ingestion:
> java.util.Date
>   
>
>
>   
>  your_coverage_name
>  
>  
> 
>  
>   
>
>
>   
>  
> regex=([0-9]{14}),format=MMddHHmmss,fullPath=true
>  TimestampFileNameExtractorSPI
>  ingestion
>   
>
>
> 
> 
> 
>  value="/path/to/directory/to/be/indexed"
> />
> 
> 
> 
> 
> 
> Usually my ImageMosaic folder structure is like this (/opt/geoserver_data
> is the GEOSERVER_DATA_DIR)
> * the indexer.xml and datastore.properties files are in a
> /opt/geoserver_data/data/
> * there is a subdirectory, usually with a date tree structure for storing
> granules from separate days in separate directories:
> /opt/geoserver_data/data//tiff/
> /MM/DD/some_file_MMDDHHmss.tiff
> * in indexer.xml I have *Recursive *parameter set to
> *true ** I usually use absolute path to the tiff directory as
> *IndexingDirectories* parameter value, e.g. /opt/geoserver_data/data/<
> IMAGEMOSAIC_DIRECTORY>/tiff
>
> Finally, if I encounter any problems during ImageMosaic creation I ensure
> to:
> * set GeoServer logging to VERBOSE
> * drop any database table created (ImageMosaic will refuse to create store
> if a table with the same name already exists), along with index and
> metadata (if using Oracle)
> * try it again and look into the log (usually the problem is not revealed
> in the last stacktrace in the log, but in some before last)
> * if it doesn't help or I still cannot figure out what is wrong I launch
> Eclipse debugger and examine what EXACTLY is wrong (I used it a lot until I
> figured out how to properly configure time parsing regexes)
>
> Regards,
>
> Peter Kovac
>
>
>
> On 29. 9. 2017 10:40, Darell van der Voort wrote:
>
> Hi Peter,
>
> This is my indexer.properties configuration:
>
> Caching=false
> TimeAttribute=ingestion
> Schema=*the_geom:Polygon,location:String,ingestion:java.util.Date:Double
> PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion)
>
> And this is my 0.properties which is automatically generated for the
> lowest zoomlevel (0):
>
> #-Automagically created from GeoTools-
> #Wed Sep 27 14:50:18 UTC 2017
> MosaicCRS=EPSG\:32617
> Levels=0.062,0.062
> Heterogeneous=false
> TimeAttribute=ingestion
> AbsolutePath=false
> Name=0
> TypeName=0
> Caching=false
> ExpandToRGB=false
> LocationAttribute=location
> SuggestedSPI=it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReaderSpi
> CheckAuxiliaryMetadata=false
> LevelsNum=1
>
> So like I said, the plugin automatically configures level 0 to the
> PostgreSQL DB if I add the datastore.properties files, but refuses to do so
> for other levels if I add the datastore.properties file to the folder
> levels. If I do not add the datastore.properties file to the additional
> levels, the plugin automatically creates a shapefile for these levels,
> while using the DB connection for level 0. This combination works, even
> with a time dimension. However I would like to use the DB connection for
> all levels and use unique table names in the DB.
>
> Kind regards,
> Darell
>
>
>
>
> On Fri, Sep 29, 2017 at 9:59 AM, Peter Kovac <
> peter.ko...@microstep-mis.com> wrote:
>
>> Hi Darell,
>>
>> can you post your indexer.xml configuration for the imagemosaic?
>>
>> Regards,
>>
>> Peter Kovac
>>
>> On 28. 9. 2017 9:59, 

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Darell van der Voort]

Hi Peter,

This is my indexer.properties configuration:

Caching=false
TimeAttribute=ingestion
Schema=*the_geom:Polygon,location:String,ingestion:java.util.Date:Double
PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion)

And this is my 0.properties which is automatically generated for the lowest
zoomlevel (0):

#-Automagically created from GeoTools-
#Wed Sep 27 14:50:18 UTC 2017
MosaicCRS=EPSG\:32617
Levels=0.062,0.062
Heterogeneous=false
TimeAttribute=ingestion
AbsolutePath=false
Name=0
TypeName=0
Caching=false
ExpandToRGB=false
LocationAttribute=location
SuggestedSPI=it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReaderSpi
CheckAuxiliaryMetadata=false
LevelsNum=1

So like I said, the plugin automatically configures level 0 to the
PostgreSQL DB if I add the datastore.properties files, but refuses to do so
for other levels if I add the datastore.properties file to the folder
levels. If I do not add the datastore.properties file to the additional
levels, the plugin automatically creates a shapefile for these levels,
while using the DB connection for level 0. This combination works, even
with a time dimension. However I would like to use the DB connection for
all levels and use unique table names in the DB.

Kind regards,
Darell




On Fri, Sep 29, 2017 at 9:59 AM, Peter Kovac 
wrote:

> Hi Darell,
>
> can you post your indexer.xml configuration for the imagemosaic?
>
> Regards,
>
> Peter Kovac
>
> On 28. 9. 2017 9:59, Darell van der Voort wrote:
>
> Hello everybody,
>
> I'm running Geoserver 2.11.2 on Java 1.8.0_144 32 bits on Amazon Linux
> together with Tomcat 9. The plugins I have installed are imagemosaic,
> imagepyramid, GeowebCache (internal) and Geofence.
>
> When using the imagemosaic plugin I am able to use a Postgresql DB for
> storing the granules and time dimension via a datastore.properties file.
> The file looks like:
>
> SPI=org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory
> schema=database_schema_name
> Loose\ bbox=true
> Estimated\ extends=false
> validate\ connections=true
> Connection\ timeout=10
> preparedStatements=true
> jndiReferenceName=java:comp/env/jdbc/postgis
>
> Which work as expected. However when I add the same file in the
> imagepyramid datastructure, this setup fails. When I add this
> datastore.properties in just the lowest level (0) then that level gets
> configured in a database table, while the higher levels are configured
> using a shapefile. When I add a datastore.properties to all levels,
> geoserver returns the following error:
>
> 2017-09-28 07:39:40,780 ERROR [gce.imagemosaic] -
> java.io.IOException
> at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.
> createGranuleCatalogFromDatastore(ImageMosaicConfigHandler.java:406)
> at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createCatalog(
> ImageMosaicConfigHandler.java:329)
> at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.buildCatalog(
> ImageMosaicConfigHandler.java:962)
> at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.indexingPreamble(
> ImageMosaicConfigHandler.java:941)
> at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$
> MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:109)
> at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(
> ImageMosaicDirectoryWalker.java:197)
> at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:460)
> at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1336)
> at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(
> ImageMosaicReader.java:532)
> at org.geotools.gce.imagemosaic.ImageMosaicReader.(
> ImageMosaicReader.java:479)
> at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(
> ImageMosaicFormat.java:504)
> at org.geotools.gce.imagepyramid.Utils.checkSource(Utils.java:200)
> at org.geotools.gce.imagepyramid.ImagePyramidReader.(
> ImagePyramidReader.java:177)
> at org.geotools.gce.imagepyramid.ImagePyramidFormat.getReader(
> ImagePyramidFormat.java:249)
> at org.geotools.gce.imagepyramid.ImagePyramidFormat.getReader(
> ImagePyramidFormat.java:56)
> at org.geoserver.catalog.ResourcePool.getGridCoverageReader(
> ResourcePool.java:1511)
> at org.geoserver.catalog.ResourcePool.getGridCoverageReader(
> ResourcePool.java:1441)
> at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(
> CoverageStoreInfoImpl.java:59)
> .
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.
> createGranuleCatalogFromDatastore(ImageMosaicConfigHandler.java:389)
> ... 160 more
>
> Is it possible to configure the granules of all levels in the database?
> And if so, how would my datastore.properties file look like? Like I said,
> it works fine for the lowest 

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Peter Kovac]

I usually configure ImageMosaic for time-series plain GeoTIFF granules 
so my advice might not be 100% accurate but I hope it helps anyway.


Firstly, your schema seems wrong to me: you have two types specified for 
ingestion - java.util.Date and a Double after that. Perhaps another 
attribute of double type is missing or the :Double part should be deleted.


Otherwise, I recommend using indexer.xml instead of indexer.properties 
for configuration. In your case it might look like this:




   
  
 ref="timeCollector">ingestion

  
   
   
  
*the_geom:Polygon,location:String,ingestion:java.util.Date
  
   
   
  
 your_coverage_name
 
 
    
 
  
   
   
  
regex=([0-9]{14}),format=MMddHHmmss,fullPath=true

TimestampFileNameExtractorSPI
 ingestion
  
   
   
    

    
    
    value="/path/to/directory/to/be/indexed" />
    
    
    

    


Usually my ImageMosaic folder structure is like this 
(/opt/geoserver_data is the GEOSERVER_DATA_DIR)
* the indexer.xml and datastore.properties files are in a 
/opt/geoserver_data/data/
* there is a subdirectory, usually with a date tree structure for 
storing granules from separate days in separate directories: 
/opt/geoserver_data/data//tiff//MM/DD/some_file_MMDDHHmss.tiff

* in indexer.xml I have *Recursive *parameter set to *true
** I usually use absolute path to the tiff directory as 
*IndexingDirectories* parameter value, e.g. 
/opt/geoserver_data/data//tiff


Finally, if I encounter any problems during ImageMosaic creation I 
ensure to:

* set GeoServer logging to VERBOSE
* drop any database table created (ImageMosaic will refuse to create 
store if a table with the same name already exists), along with index 
and metadata (if using Oracle)
* try it again and look into the log (usually the problem is not 
revealed in the last stacktrace in the log, but in some before last)
* if it doesn't help or I still cannot figure out what is wrong I launch 
Eclipse debugger and examine what EXACTLY is wrong (I used it a lot 
until I figured out how to properly configure time parsing regexes)


Regards,

Peter Kovac



On 29. 9. 2017 10:40, Darell van der Voort wrote:

Hi Peter,

This is my indexer.properties configuration:

Caching=false
TimeAttribute=ingestion
Schema=*the_geom:Polygon,location:String,ingestion:java.util.Date:Double
PropertyCollectors=TimestampFileNameExtractorSPI[timeregex](ingestion)

And this is my 0.properties which is automatically generated for the 
lowest zoomlevel (0):


#-Automagically created from GeoTools-
#Wed Sep 27 14:50:18 UTC 2017
MosaicCRS=EPSG\:32617
Levels=0.062,0.062
Heterogeneous=false
TimeAttribute=ingestion
AbsolutePath=false
Name=0
TypeName=0
Caching=false
ExpandToRGB=false
LocationAttribute=location
SuggestedSPI=it.geosolutions.imageioimpl.plugins.tiff.TIFFImageReaderSpi
CheckAuxiliaryMetadata=false
LevelsNum=1

So like I said, the plugin automatically configures level 0 to the 
PostgreSQL DB if I add the datastore.properties files, but refuses to 
do so for other levels if I add the datastore.properties file to the 
folder levels. If I do not add the datastore.properties file to the 
additional levels, the plugin automatically creates a shapefile for 
these levels, while using the DB connection for level 0. This 
combination works, even with a time dimension. However I would like to 
use the DB connection for all levels and use unique table names in the DB.


Kind regards,
Darell




On Fri, Sep 29, 2017 at 9:59 AM, Peter Kovac 
> 
wrote:


Hi Darell,

can you post your indexer.xml configuration for the imagemosaic?

Regards,

Peter Kovac


On 28. 9. 2017 9:59, Darell van der Voort wrote:

Hello everybody,

I'm running Geoserver 2.11.2 on Java 1.8.0_144 32 bits on Amazon
Linux together with Tomcat 9. The plugins I have installed are
imagemosaic, imagepyramid, GeowebCache (internal) and Geofence.

When using the imagemosaic plugin I am able to use a Postgresql
DB for storing the granules and time dimension via a
datastore.properties file. The file looks like:

SPI=org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory
schema=database_schema_name
Loose\ bbox=true
Estimated\ extends=false
validate\ connections=true
Connection\ timeout=10
preparedStatements=true
jndiReferenceName=java:comp/env/jdbc/postgis

Which work as expected. However when I add the same file in the
imagepyramid datastructure, this setup fails. When I add this
datastore.properties in just the lowest level (0) then that level
gets configured in a database table, while the higher levels are
configured using a shapefile. When I add a datastore.properties
to all levels, geoserver returns the following error:

2017-09-28 07:39:40,780 ERROR [gce.imagemosaic] -

Re: [Geoserver-users] ImagePyramid using Postgresql for granules

[Peter Kovac]

Hi Darell,

can you post your indexer.xml configuration for the imagemosaic?

Regards,

Peter Kovac


On 28. 9. 2017 9:59, Darell van der Voort wrote:

Hello everybody,

I'm running Geoserver 2.11.2 on Java 1.8.0_144 32 bits on Amazon Linux 
together with Tomcat 9. The plugins I have installed are imagemosaic, 
imagepyramid, GeowebCache (internal) and Geofence.


When using the imagemosaic plugin I am able to use a Postgresql DB for 
storing the granules and time dimension via a datastore.properties 
file. The file looks like:


SPI=org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory
schema=database_schema_name
Loose\ bbox=true
Estimated\ extends=false
validate\ connections=true
Connection\ timeout=10
preparedStatements=true
jndiReferenceName=java:comp/env/jdbc/postgis

Which work as expected. However when I add the same file in the 
imagepyramid datastructure, this setup fails. When I add this 
datastore.properties in just the lowest level (0) then that level gets 
configured in a database table, while the higher levels are configured 
using a shapefile. When I add a datastore.properties to all levels, 
geoserver returns the following error:


2017-09-28 07:39:40,780 ERROR [gce.imagemosaic] -
java.io.IOException
at 
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createGranuleCatalogFromDatastore(ImageMosaicConfigHandler.java:406)
at 
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createCatalog(ImageMosaicConfigHandler.java:329)
at 
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.buildCatalog(ImageMosaicConfigHandler.java:962)
at 
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.indexingPreamble(ImageMosaicConfigHandler.java:941)
at 
org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:109)
at 
org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:197)

at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:460)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1336)
at 
org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:532)
at 
org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:479)
at 
org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:504)

at org.geotools.gce.imagepyramid.Utils.checkSource(Utils.java:200)
at 
org.geotools.gce.imagepyramid.ImagePyramidReader.(ImagePyramidReader.java:177)
at 
org.geotools.gce.imagepyramid.ImagePyramidFormat.getReader(ImagePyramidFormat.java:249)
at 
org.geotools.gce.imagepyramid.ImagePyramidFormat.getReader(ImagePyramidFormat.java:56)
at 
org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1511)
at 
org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1441)
at 
org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)

.
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at 
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createGranuleCatalogFromDatastore(ImageMosaicConfigHandler.java:389)

... 160 more

Is it possible to configure the granules of all levels in the 
database? And if so, how would my datastore.properties file look like? 
Like I said, it works fine for the lowest level (0) but it fails for 
all the levels above. If it may not be possible, is it sufficient to 
use the shapefiles as an index for the granules? The geoserver is used 
in a production environment and I am worried about performance issues 
when using the shapefiles as index.


Furthermore the imagepyramid automatically names the table after the 
level folder (0). This folder and table name are automatically created 
when configuring the datastore. Is there a way to custom name the 
folder and thus the table name? Otherwise I could end up with multiple 
'0' tables in the same schema.


Many thanks!

Kind regards,
Darell


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this 
list:
- Earning your support instead of buying it, but Ian Turton: 
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: 
http://geoserver.org/comm/userlist-guidelines.html

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users


--
Peter Kovac
IMS Programmer
MicroStep-MIS
peter.ko...@microstep-mis.com

--
Check out the vibrant tech community 

[Geoserver-users] ImagePyramid using Postgresql for granules

[Darell van der Voort]

Hello everybody,

I'm running Geoserver 2.11.2 on Java 1.8.0_144 32 bits on Amazon Linux
together with Tomcat 9. The plugins I have installed are imagemosaic,
imagepyramid, GeowebCache (internal) and Geofence.

When using the imagemosaic plugin I am able to use a Postgresql DB for
storing the granules and time dimension via a datastore.properties file.
The file looks like:

SPI=org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory
schema=database_schema_name
Loose\ bbox=true
Estimated\ extends=false
validate\ connections=true
Connection\ timeout=10
preparedStatements=true
jndiReferenceName=java:comp/env/jdbc/postgis

Which work as expected. However when I add the same file in the
imagepyramid datastructure, this setup fails. When I add this
datastore.properties in just the lowest level (0) then that level gets
configured in a database table, while the higher levels are configured
using a shapefile. When I add a datastore.properties to all levels,
geoserver returns the following error:

2017-09-28 07:39:40,780 ERROR [gce.imagemosaic] -
java.io.IOException
at
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createGranuleCatalogFromDatastore(ImageMosaicConfigHandler.java:406)
at
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createCatalog(ImageMosaicConfigHandler.java:329)
at
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.buildCatalog(ImageMosaicConfigHandler.java:962)
at
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.indexingPreamble(ImageMosaicConfigHandler.java:941)
at
org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:109)
at
org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:197)
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:460)
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1336)
at
org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:532)
at
org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:479)
at
org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:504)
at org.geotools.gce.imagepyramid.Utils.checkSource(Utils.java:200)
at
org.geotools.gce.imagepyramid.ImagePyramidReader.(ImagePyramidReader.java:177)
at
org.geotools.gce.imagepyramid.ImagePyramidFormat.getReader(ImagePyramidFormat.java:249)
at
org.geotools.gce.imagepyramid.ImagePyramidFormat.getReader(ImagePyramidFormat.java:56)
at
org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1511)
at
org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1441)
at
org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)
.
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.createGranuleCatalogFromDatastore(ImageMosaicConfigHandler.java:389)
... 160 more

Is it possible to configure the granules of all levels in the database? And
if so, how would my datastore.properties file look like? Like I said, it
works fine for the lowest level (0) but it fails for all the levels above.
If it may not be possible, is it sufficient to use the shapefiles as an
index for the granules? The geoserver is used in a production environment
and I am worried about performance issues when using the shapefiles as
index.

Furthermore the imagepyramid automatically names the table after the level
folder (0). This folder and table name are automatically created when
configuring the datastore. Is there a way to custom name the folder and
thus the table name? Otherwise I could end up with multiple '0' tables in
the same schema.

Many thanks!

Kind regards,
Darell
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this 
list:
- Earning your support instead of buying it, but Ian Turton: 
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: 
http://geoserver.org/comm/userlist-guidelines.html

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users