I’m not sure what the issue is. I edited my timeregex.propeties file to
include the format, and correct regex. I also updated directory to remove the
ascii files, to have the correct timeregex in the tiff filename, and I still
get issues with the indexer.properties file. I traced the geotools code and
heres what ive found:
The code throws an error in the AciiGridsImageReaderSpi.class when it tries to
decode the indexer.properties file as AsciiGridRasterType and parse the header,
line 215.
// Now, I have an ImageInputStream and I can try to see if input
can be
// decoded by doing header parsing
//
/////////////////////////////////////////////////////////////////////
try {
// Header Parsing to check if it is an EsriAsciiGridRaster
asciiRaster =
AsciiGridRaster.AsciiGridRasterType.ESRI.createAsciiGridRaster(spiImageInputStream);
asciiRaster.parseHeader();
I’m not sure why the ImageMosaicWalker class is even trying to read in
*.properties files. If I add a line of code in the ImageMosaicWalker class to
skip files with properties or xml extensions then everything works fine. Maybe
I’m using a different version of geotools, or have different plugins then what
you are testing with but I cant get it to work without that “skip” line of code.
Thanks for your help
Dominique
From: [email protected] [mailto:[email protected]] On Behalf Of
Daniele Romagnoli
Sent: Wednesday, June 17, 2015 2:21 AM
To: Bessette-Halsema, Dominique E
Cc: Mike Grogan; [email protected]
Subject: Re: [Geoserver-users] image mosaic with GeoServerRESTPublisher
Hi Dominique,
I was able to configure an imageMosaic on your sample dataset without changing
any line of code.
A couple of notes on the issues you had.
Why are you using both asc and TIFF files on the mosaic?
Note that your ascii files aren't properly formed. They define a NO_DATA_VALUE
field in the header whilst it should be NODATA_VALUE [1].
This results into the reader being unable to parse it. Also note that they
don't have CoordinateReferenceSystem definition so you should consider adding a
PRJ beside them.
Once removing them from the mosaic folder, I found the other issue:
Your time regex isn't matching the times defined in the filename.
If your file name is "133_2015-06-12T03_00_00.000Z.tif", then the timeregex you
specified won't match that (I'm reporting my notes based on the files you have
attached to the email):
regex =
[0-9]{8}T[0-9]{9}Z(\\?!.\\*[0-9]{8}T[0-9]{9}Z.\\*<file:///\\%3f!.\*%5b0-9%5d%7b8%7dT%5b0-9%5d%7b9%7dZ.\*>)
I have renamed your files as:
133_20150612T000000Z.tif which contains an ISO8601 time format and specified
that regex:
regex=[0-9]{8}T[0-9]{6}Z
Which worked.
Alternatively, if you can't change your file names, you can instruct the time
parser to use a custom format.
As an instance, in order to deal with your attached files:
133_2015-06-12T03_00_00.000Z.tif
you may set this content into the timeregex.properties:
regex=[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}_[0-9]{2}_[0-9]{2}.[0-9]{3}Z,format=yyyy-MM-dd'T'HH_mm_ss.SSS'Z'
Note the regex is containing time split into pieces, due to the presence of
dashes and underscores on your file name.
Then, there is a "format=yyyy-MM-dd'T'HH_mm_ss.SSS'Z'" section in order to
instruct the parser that the time is formatted that way, so 4 digits for the
year, a dash sign, 2 digits for the month, a dash sign............. then a T
char, 2 digits for the hour, an underscore and so on.
The "format" option can be added to a time regex property to deal with times
which aren't ISO8601 formatted.
Hope this helps.
Please, let us know if you still have issues.
Cheers,
Daniele
[1]: https://en.wikipedia.org/wiki/Esri_grid#ASCII
On Tue, Jun 16, 2015 at 7:10 PM, Bessette-Halsema, Dominique E
<[email protected]<mailto:[email protected]>> wrote:
I looked at the geotools code in the gt-imagemosaic module and the
GridFormatFinder is reading the indexer.properties file as a NetCDF file. I
added the line logExcludes.contains(extension) to the If check in the
ImageMosaicWalker class line 156 and that fixed that issue. I also had to use
the indexer.xml file instead of the indexer.properties file otherwise the
customformat object in the TimestampFileNameExtractor is null and another error
was thrown.
ImageMosaicWalker.java
if (logExcludes.contains(extension) || (format instanceof
UnknownFormat) || format == null) {
if (!logExcludes.contains(extension)) {
eventHandler.fireFileEvent(Level.INFO, fileBeingProcessed,
false, "Skipped file "
+ fileBeingProcessed + ": File format is not
supported.",
((fileIndex * 99.0) / numFiles));
}
return;
}
I also had to update the org.geotools.data.DataUtilities class createType
method because it was parsing my feature name incorrectly, but I am assuming
that’s a naming convention I need to fix.
After these simple updates I can now see the correct attributes in the gis
table, and the time dimension in the get capabilities document.
Dom
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]<mailto:[email protected]>] On Behalf Of
Daniele Romagnoli
Sent: Tuesday, June 16, 2015 9:02 AM
To: Bessette-Halsema, Dominique E
Cc: Mike Grogan;
[email protected]<mailto:[email protected]>
Subject: Re: [Geoserver-users] image mosaic with GeoServerRESTPublisher
Hi Dominique,
I'll also take a look on this during the week to figure out what is going on.
Anyway, note that if you remove the indexer properties file to let the
ImageMosaic work, the TIME dimension won't be taken into account as expected.
There is no way to have the time dimension properly set if the indexer is
unavailable since it's the indexer itself which contains instructions for the
imageMosaic on how to retrieve the time values. Wondering if there is any
problem in parsing the regexes you have specified.
Hope this helps.
Cheers,
Daniele.
On Tue, Jun 16, 2015 at 5:31 PM, Bessette-Halsema, Dominique E
<[email protected]<mailto:[email protected]>> wrote:
I’m going through the geotools code to figure it out. If I come up with a
solution I’ll post it
Thanks
Dom
From: Mike Grogan
[mailto:[email protected]<mailto:[email protected]>]
Sent: Tuesday, June 16, 2015 8:04 AM
To: Bessette-Halsema, Dominique E
Cc:
[email protected]<mailto:[email protected]>
Subject: Re: [Geoserver-users] image mosaic with GeoServerRESTPublisher
Dominique,
I'll do my best to take a look at what you have sent -- and see if I can load
it on a system -- when I can ... but I'm just another fellow user.
- Mike
On Mon, Jun 15, 2015 at 4:32 PM, Bessette-Halsema, Dominique E
<[email protected]<mailto:[email protected]>> wrote:
Hi Mike
I added the tiff and properties files from the example in the below tutorial
onto my machine, and I have the same issue. I can create the database table,
and datastore if I remove the indexer.properties file but then I can’t add the
time and elevation dimensions. When I add the indexer.properties file I see
the errors in the attached log file.
Here’s the table definition
CREATE TABLE products."temperatureLZWdataset"
(
fid serial NOT NULL,
location character varying(255),
the_geom geometry,
CONSTRAINT "temperatureLZWdataset_pkey" PRIMARY KEY (fid),
CONSTRAINT enforce_dims_the_geom CHECK (st_ndims(the_geom) = 2),
CONSTRAINT enforce_geotype_the_geom CHECK (geometrytype(the_geom) =
'POLYGON'::text OR the_geom IS NULL),
CONSTRAINT enforce_srid_the_geom CHECK (st_srid(the_geom) = 4326)
)
I attached the properties files.
Also the file names in the tutorial are in the format
gfs50kmTemperature20130310T180000000Z_0200.000_.tiff but the tutorial states
“Remember that every tif file must follow this naming convention:
{coveragename}_{timestamp}_[{elevation}].tif” Shouldn’t the tiff name be
“gfs50kmTemperature_20130310T180000000Z_0200.000_.tiff” ?
Thanks
Dominique
From: Mike Grogan
[mailto:[email protected]<mailto:[email protected]>]
Sent: Monday, June 15, 2015 9:37 AM
To: Bessette-Halsema, Dominique E
Cc: Daniele Romagnoli;
[email protected]<mailto:[email protected]>
Subject: Re: [Geoserver-users] image mosaic with GeoServerRESTPublisher
Regarding specifically the ImageMosaicDirectoryWalker errors, I am thinking
this is an issue with the specification of times in your filenames and the
regular expression used against that. I have found that I usually have to more
fully specify the time with hours, minutes, seconds after the 'T' ... as
GeoServer is designed to recognize just some specific ISO forms of time, if I
recall.
Where you have 133_2015-06-11T03.tif, you might try 133_2015-06-11T030000Z.tif
and a regex of regex = _[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{6}Z, or something
similar to the filename and regex patterns given in
http://docs.geoserver.org/latest/en/user/tutorials/imagemosaic_timeseries/imagemosaic_time-elevationseries.html.
Whenever I hit a ImageMosaicDirectoryWalker error, it seems like it has usually
been due to my definition of time in the filename and regex.
If the .asc files are giving other errors, you might try this out with just the
.tif files first and see if it works when expanding the definition of time in
the filename and the regex.
- Mike
On Mon, Jun 15, 2015 at 11:54 AM, Bessette-Halsema, Dominique E
<[email protected]<mailto:[email protected]>> wrote:
Thanks for the reply. The image mosaic does not work in the console either. I
see the following error.
Could not list layers for this store, an error occurred retrieving them: Failed
to create reader from
file:data/dynamic/isisgridcoverage/ThresholdingMissionID-2/coaID-4/RuleID-5/ThresholdID-133
and hints null
I did reply to my original email stating that I could get the Image Mosaic to
work if I remove the indexer.properties file but I can’t get the TIME dimension
to work. I attached the email. Please review it.
Thank you
Dominique
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]<mailto:[email protected]>] On Behalf Of
Daniele Romagnoli
Sent: Monday, June 15, 2015 1:05 AM
To: Bessette-Halsema, Dominique E
Cc:
[email protected]<mailto:[email protected]>
Subject: Re: [Geoserver-users] image mosaic with GeoServerRESTPublisher
Hi Dominique,
a couple of feedbacks for your logs.
Please, read below...
On Fri, Jun 12, 2015 at 2:46 AM, Bessette-Halsema, Dominique E
<[email protected]<mailto:[email protected]>> wrote:
Is the image mosaic the best way to add TIME and ELEVATION dimensions to a
layer in geoserver?
If so, I’m having an issues creating an image mosaic with the
GeoServerRESTPublisher API in GeoServer 2.7.
I create several geotiffs to a file directory within the
GEOSERVER_DATA_DIR/data/. with the name structure ID_TIME.tiff
(133_2015-06-11T03.tif)
I then create the indexer, timeregex and datastore properties files.
Using the GeoServerRESTPublisher I try to post the mosaic to geoserver
publishExternalMosaic(workspace, storename, new File(path), coverageEnc,
encoder)
It creates the ImageMosaic Store but fails when it creates the layer
I get the following error:
00:19:40,009 INFO [org.geoserver.catalog.rest] (http-/0.0.0.0:8080-2) Using
existing coverage store: COA_ID_1.RULE_ID_5.THRESHOLD_ID_133
ERROR 4:
`/gis/geoserver/external/data/dynamic/missionthresholding/THRESHOLDING_MISSION_ID_1/COA_ID_1/RULE_ID_5/THRESHOLD_ID_133/indexer.properties'
not recognised as a supported file format.
ERROR 4:
`/gis/geoserver/external/data/dynamic/missionthresholding/THRESHOLDING_MISSION_ID_1/COA_ID_1/RULE_ID_5/THRESHOLD_ID_133/indexer.properties'
not recognised as a supported file format.
Not real errors... they are INFO messages. The GDAL machinery is reporting that
it's unable to open the indexer.properties file using a
GDAL driver.
00:19:40,063 ERROR [class org.geotools.gce.imagemosaic.Utils]
(http-/0.0.0.0:8080-2) For input string: "E": java.lang.NumberFormatException:
For input string: "E"
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250)
[rt.jar:1.7.0_72]
at java.lang.Double.parseDouble(Double.java:540) [rt.jar:1.7.0_72]
at
it.geosolutions.imageio.utilities.StringToDouble.compute(StringToDouble.java:272)
[imageio-ext-utilities-1.1.10.jar:]
at
it.geosolutions.imageio.plugins.arcgrid.raster.AsciiGridRaster.getValue(AsciiGridRaster.java:1154)
[imageio-ext-arcgrid-1.1.10.jar:]
at
it.geosolutions.imageio.plugins.arcgrid.raster.EsriAsciiGridRaster.parseHeader(EsriAsciiGridRaster.java:161)
[imageio-ext-arcgrid-1.1.10.jar:]
at
it.geosolutions.imageio.plugins.arcgrid.spi.AsciiGridsImageReaderSpi.canDecodeInput(AsciiGridsImageReaderSpi.java:217)
[imageio-ext-arcgrid-1.1.10.jar:]
…….
It seems some errors occurred on parsing the ASCii file's headers. Do you
have any chance to share one of these files? I'm wondering whether their
Headers are properly formed.
00:29:08,039 WARN [org.geotools.gce.imagemosaic] (http-/0.0.0.0:8080-2)
Failure occurred while collecting the granules:
org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled
at
org.apache.commons.io.DirectoryWalker.checkIfCancelled(DirectoryWalker.java:395)
[commons-io-2.1.jar:2.1]
at
org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:368)
[commons-io-2.1.jar:2.1]
at
org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)
[commons-io-2.1.jar:2.1]
at
org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.<init>(ImageMosaicDirectoryWalker.java:114)
[gt-imagemosaic-13.1.jar:]
at
org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)
[gt-imagemosaic-13.1.jar:]
at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:354)
[gt-imagemosaic-13.1.jar:]
at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1223)
[gt-imagemosaic-13.1.jar:]
at
org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:516)
[gt-imagemosaic-13.1.jar:]
at
org.geotools.gce.imagemosaic.ImageMosaicReader.<init>(ImageMosaicReader.java:468)
[gt-imagemosaic-13.1.jar:]
at
org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)
[gt-imagemosaic-13.1.jar:]
at
org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)
[gt-imagemosaic-13.1.jar:
As suggested by Mike, could you also check if you are able to configure it
through the GeoServer GUI by simply configuring an ImageMosaic store pointing
to that MOSAIC_DATA_DIR folder?
Please, let us know.
Best Regards,
Daniele
more INFO:
MOSAIC_DATA_DIR contents:
133_2015-06-11T00.asc
133_2015-06-11T01.asc
133_2015-06-11T02.asc
133_2015-06-11T03.asc
133_2015-06-11T00.tif
133_2015-06-11T01.tif
133_2015-06-11T02.tif
133_2015-06-11T03.tif
indexer.properties
datastore.properties
timeregex.properties
datastore.properties
schema = geoimpact
preparedStatements = true
SPI = org.geotools.data.postgis.PostgisNGJNDIDataStoreFactory
jndiReferenceName = java:jboss/datasources/gisDB
indexer.properties:
TimeAttribute = validtime
AdditionalDomainAttributes = analysis_time
Schema =
*the_geom:Polygon,location:String,validtime:java.util.Date,analysis_time:java.util.Date
PropertyCollectors =
TimestampFileNameExtractorSPI[timeregex](validtime),TimestampFileNameExtractorSPI[timeregex](analysis_time)
Timeregex.properties
regex = [0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}
Thanks for you help
Dom
------------------------------------------------------------------------------
_______________________________________________
Geoserver-users mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==
Ing. Daniele Romagnoli
Senior Software Engineer
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313<tel:%2B39%200584%20962313>
fax: +39 0584 1660272<tel:%2B39%200584%201660272>
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo
è consentito esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro
sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act (Legislative
Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in
accord with its purpose, any disclosure, reproduction, copying, distribution,
or either dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the intended
recipient, please contact immediately the sender by telephone, fax or e-mail
and delete the information in this message that has been received in error. The
sender does not give any warranty or accept liability as the content, accuracy
or completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.
------------------------------------------------------------------------------
_______________________________________________
Geoserver-users mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/geoserver-users
--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==
Ing. Daniele Romagnoli
Senior Software Engineer
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313<tel:%2B39%200584%20962313>
fax: +39 0584 1660272<tel:%2B39%200584%201660272>
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo
è consentito esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro
sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act (Legislative
Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in
accord with its purpose, any disclosure, reproduction, copying, distribution,
or either dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the intended
recipient, please contact immediately the sender by telephone, fax or e-mail
and delete the information in this message that has been received in error. The
sender does not give any warranty or accept liability as the content, accuracy
or completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.
--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==
Ing. Daniele Romagnoli
Senior Software Engineer
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo
è consentito esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro
sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act (Legislative
Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in
accord with its purpose, any disclosure, reproduction, copying, distribution,
or either dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the intended
recipient, please contact immediately the sender by telephone, fax or e-mail
and delete the information in this message that has been received in error. The
sender does not give any warranty or accept liability as the content, accuracy
or completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.
------------------------------------------------------------------------------
_______________________________________________
Geoserver-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
