This all looks reasonable.
I also think I prefer a single main file with a list of supporting files,
it makes it easier to figure out exactly which supporting files are
associated with each main file.
To handle the latter, I'm almost wanting a getDataFiles(File mainFiles),
> getSupportFiles(File mainFile)
> to be added to the interfaces... which are starting to get a little
> crowded.
> As an alternative, have getFiles(File mainFile), and if it's called with
> null, we get the main list of files,
> if called with a file, the sidecar data files, and likewise for
> getSupportFiles(File mainFiles), main
> support files if a null argument is passed, support files or the main file
> if a non null argument is passed.
>
Based on this description, it almost looks as if another option could be:
class FileGroup {
Map(File, List<File>) files;
Map<String,Object> metadata;
}
Then FileGroup.getDataFiles() returns files.keySet() and FileGroup.
getSupportFiles(mainFile) returns files.get(mainFile).
I am not sure how good of an idea this is though?
Bringing up Importer again, once FileGroup gets imported and is stable, it
might be nice to modify org.geoserver.importer.SpatialFile to use or extend
it for consistencies sake.
Torben
On Tue, Oct 6, 2015 at 6:09 AM, Daniele Romagnoli <
daniele.romagn...@geo-solutions.it> wrote:
> Ok. I have checked the code and updated it a bit when discovering a couple
> of implementation issues.
>
> Some updates commenting original Andrea's notes.
>
> * A mosaic is made of files, main files, but those files might have
>>> sidecar files too, like, it can
>>> be made of world image files... when one downloads the main file, its
>>> sidecar files need to
>>> go along, but we cannot list the sidecar files in the main lists, we
>>> need to associate them with
>>> the particular file we want to download.
>>>
>>> To handle the latter, I'm almost wanting a getDataFiles(File mainFiles),
>>> getSupportFiles(File mainFile)
>>> to be added to the interfaces... which are starting to get a little
>>> crowded.
>>> As an alternative, have getFiles(File mainFile), and if it's called with
>>> null, we get the main list of files,
>>> if called with a file, the sidecar data files, and likewise for
>>> getSupportFiles(File mainFiles), main
>>> support files if a null argument is passed, support files or the main
>>> file if a non null argument is passed.
>>>
>>> Or... we stop returning File and we return a FileGroup class that would
>>> contain:
>>>
>>> class FileGroup {
>>> List<File> main;
>>> List<File> support;
>>> }
>>>
>>
> I would simply base a group on top of a main files + support files (if
> any). Therefore:
>
> class FileGroup {
> File main;
> List<File> support;
> Map<String,Object> metadata;
> }
>
> Note: I would also add a metadata map providing info on the Group. Let's
> think about a GRIB main file covering a time range T0-T1 and an elevation
> range Z0-Z1. Adding this kind of info in the metadata map would allow
> recognizing the domain of validity of the group.
>
>
>> Hi,
>>> I've been trying to hash out the design for this stuff a bit using code,
>>> here
>>> is a preliminary result:
>>>
>>> https://github.com/aaime/geotools/commit/15dec9e635b5a23db6d230972eca5e566835fe3f
>>>
>>> Implementing it made me think a bit about the current interfaces, and
>>> ways to improve them.
>>> The interfaces work nicely with geotiff/netcdf/grib/worldimage, not as
>>> well with imagemosaic.
>>>
>>> Issues:
>>> * A image mosaic can be made of hundred of thousands of files, I guess
>>> we might want
>>> to return a CloseableIterator to gracefully handle that case
>>> (CloseableIterator would be
>>> a new interface, extending AutoCloseable and Iterator).
>>>
>> * A image mosaic has multiple coverage, how about adding a ResourceInfo
>>> getInfo(Coverage) to
>>> GridCoverageReader and only return list of actual data files for that
>>> one, and return
>>> the mosaic index/properties/config files for the service info ones?
>>>
>>
> Agreed with the getInfo(coverageName), getInfo() new methods to be added
> to the GridCoverage2DReader interface.
> The first one returning a ResourceInfo, the second one returning a
> ServiceInfo.
>
> ResourceInfo and ServiceInfo interfaces will be extended by
> FileResourceInfo and FileServiceInfo which have a new method getFiles.
> returning a new FileGroupProvider interface instead of a simple
> List<FileGroup>.
>
> FileGroupProvider interface has a method getFiles(Query) returning a
> CloseableIterator<FileGroup>.
> The query object can be used to return a reduced set of files (as an
> instance, matching a specific location).
> A null query means retrieving all available files groups.
>
> That CloseableIterator is already defined in both shapefile module and
> GeoServer catalog. We can move that interface to core libs (gt-api) and
> deprecate the others in favor of this.
> Moreover, returning the iterator instead of a pure List of files allows
> the caller to call "close" on the iterator when done.
>
>
>>
>>>
>>
> This set of changes are currently available at
> https://github.com/dromagnoli/geotools/tree/rawdownload
>
> I'm going to create separate threads/proposals/JIRA/pullrequests for the
> CloseableIterator topic, and GridCoverage2DReader new methods and related
> interfaces, leaving the reader specific implementation into a separate PR.
>
> Feedbacks?
>
> Cheers,
> Daniele
>
>
>
>
>
>>
>>>
>>> Opinions?
>>>
>>> Cheers
>>> Andrea
>>>
>>>
>>> On Thu, May 14, 2015 at 2:23 PM, Andrea Aime <
>>> andrea.a...@geo-solutions.it> wrote:
>>>
>>>> On Wed, May 13, 2015 at 7:47 PM, Jody Garnett <jody.garn...@gmail.com>
>>>> wrote:
>>>>
>>>>> You mean, sub-interfaces of ServiceInfo and ResourceInfo, right?
>>>>>>
>>>>>
>>>>> I do.
>>>>>
>>>>> interface FileServiceInfo extends ServiceInfo {
>>>>> /**
>>>>> * File or directory providing service content.
>>>>> * Optional information provided to facilitate moving, renaming to
>>>>> deleting service contents on disk.
>>>>> *
>>>>> * For services specified by a single directory location check
>>>>> individual ResourceInfo.getFiles() for more information.
>>>>> *
>>>>> * @return files or directory locating service content.
>>>>> List<File> getFiles();
>>>>> }
>>>>>
>>>>> interface FileResourceInfo extends ResourceInfo {
>>>>> /**
>>>>> * File or directory providing resource content.
>>>>> * Optional information provided to facilitate moving, renaming or
>>>>> deleting resource contents on disk.
>>>>> *
>>>>> * For resources specified using a directory (such as a tileset)
>>>>> the entire directory contents are considered to define the resource.
>>>>> * @return files or directory providing resource content
>>>>> */
>>>>> List<Files> getFiles();
>>>>> }
>>>>>
>>>>
>>>> Hi Jody,
>>>> do you mind if we also add a:
>>>>
>>>> /**
>>>> * File or directory representing support files that do not qualify
>>>> as data itself (e.g. indexes, caches) that would
>>>> * still have to be dropped along with the data in case of removal
>>>> */
>>>> List<File> getSupportFiles();
>>>>
>>>> Cheers
>>>> Andrea
>>>>
>>>> --
>>>> ==
>>>> GeoServer Professional Services from the experts! Visit
>>>> http://goo.gl/NWWaa2 for more information.
>>>> ==
>>>>
>>>> Ing. Andrea Aime
>>>> @geowolf
>>>> Technical Lead
>>>>
>>>> GeoSolutions S.A.S.
>>>> Via Poggio alle Viti 1187
>>>> 55054 Massarosa (LU)
>>>> Italy
>>>> phone: +39 0584 962313
>>>> fax: +39 0584 1660272
>>>> mob: +39 339 8844549
>>>>
>>>> http://www.geo-solutions.it
>>>> http://twitter.com/geosolutions_it
>>>>
>>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>>
>>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>>> principi dettati dal D.Lgs. 196/2003.
>>>>
>>>>
>>>>
>>>> The information in this message and/or attachments, is intended solely
>>>> for the attention and use of the named addressee(s) and may be confidential
>>>> or proprietary in nature or covered by the provisions of privacy act
>>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>>> copying, distribution, or either dissemination, either whole or partial, is
>>>> strictly forbidden except previous formal approval of the named
>>>> addressee(s). If you are not the intended recipient, please contact
>>>> immediately the sender by telephone, fax or e-mail and delete the
>>>> information in this message that has been received in error. The sender
>>>> does not give any warranty or accept liability as the content, accuracy or
>>>> completeness of sent messages and accepts no responsibility for changes
>>>> made after they were sent or for other risks which arise as a result of
>>>> e-mail transmission, viruses, etc.
>>>>
>>>> -------------------------------------------------------
>>>>
>>>
>>>
>>>
>>> --
>>> ==
>>> Meet us at the INSPIRE Conference in Lisbon 25-29 May 2015! Visit
>>> http://goo.gl/WHKDXT for more information.
>>> ==
>>>
>>> Ing. Andrea Aime
>>> @geowolf
>>> Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via Poggio alle Viti 1187
>>> 55054 Massarosa (LU)
>>> Italy
>>> phone: +39 0584 962313
>>> fax: +39 0584 1660272
>>> mob: +39 339 8844549
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>
>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>> principi dettati dal D.Lgs. 196/2003.
>>>
>>>
>>>
>>> The information in this message and/or attachments, is intended solely
>>> for the attention and use of the named addressee(s) and may be confidential
>>> or proprietary in nature or covered by the provisions of privacy act
>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>> copying, distribution, or either dissemination, either whole or partial, is
>>> strictly forbidden except previous formal approval of the named
>>> addressee(s). If you are not the intended recipient, please contact
>>> immediately the sender by telephone, fax or e-mail and delete the
>>> information in this message that has been received in error. The sender
>>> does not give any warranty or accept liability as the content, accuracy or
>>> completeness of sent messages and accepts no responsibility for changes
>>> made after they were sent or for other risks which arise as a result of
>>> e-mail transmission, viruses, etc.
>>>
>>> -------------------------------------------------------
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> One dashboard for servers and applications across Physical-Virtual-Cloud
>>> Widest out-of-the-box monitoring support with 50+ applications
>>> Performance metrics, stats and reports that give you Actionable Insights
>>> Deep dive visibility with transaction tracing using APM Insight.
>>> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
>>> _______________________________________________
>>> GeoTools-Devel mailing list
>>> GeoTools-Devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>>
>>>
>>
>>
>> --
>> ==
>> GeoServer Professional Services from the experts! Visit
>> http://goo.gl/it488V for more information.
>> ==
>>
>> Ing. Daniele Romagnoli
>> Senior Software Engineer
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054 Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax: +39 0584 1660272
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> -------------------------------------------------------
>>
>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>> principi dettati dal D.Lgs. 196/2003.
>>
>>
>>
>> The information in this message and/or attachments, is intended solely
>> for the attention and use of the named addressee(s) and may be confidential
>> or proprietary in nature or covered by the provisions of privacy act
>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>> copying, distribution, or either dissemination, either whole or partial, is
>> strictly forbidden except previous formal approval of the named
>> addressee(s). If you are not the intended recipient, please contact
>> immediately the sender by telephone, fax or e-mail and delete the
>> information in this message that has been received in error. The sender
>> does not give any warranty or accept liability as the content, accuracy or
>> completeness of sent messages and accepts no responsibility for changes
>> made after they were sent or for other risks which arise as a result of
>> e-mail transmission, viruses, etc.
>>
>>
>
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Daniele Romagnoli
> Senior Software Engineer
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
------------------------------------------------------------------------------
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
