Re: [Geotools-devel] ImageMosaic API refactor proposal

[Simone Giannecchini]

I am not too fond on this name but as a general rule of thumb I always
suggest to refrain from the renaming attitude unless the original name
is _really_ bad.

This is also important as we would have in some cases to keep the old
one around if it was a public class or interface.
Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Tue, Jun 14, 2016 at 8:20 PM, Devon Tucker  wrote:
> Jody I'm not sure what a good name for the class is. Its chief
> responsibilities (before this proposal) were collecting attributes from the
> incoming granule, creating its new feature and updating the GranuleStore,
> creating the index schema, and instantiating the GranuleCatalog from the
> reader configuration. Now that I type all that out I don't think
> CatalogManager is that bad a name?
>
> On Tue, Jun 14, 2016 at 10:42 AM, Jody Garnett 
> wrote:
>>
>> I note the prior proposal had CatalogManager renamed to
>> GranuleCatalogManager is that still appropriate?
>>
>> Will you be removing the original proposal then?
>>
>> --
>> Jody Garnett
>>
>> On 9 June 2016 at 16:39, Devon Tucker  wrote:
>>>
>>> Thanks for all the feedback. Like Jody said, we're probably going to
>>> break the proposal apart a bit. I've started pulling apart the index/catalog
>>> management into its own proposal:
>>>
>>>
>>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-Index-Catalog-management-for-improved-extensibility
>>>
>>> Still a work in progress. Will work on adding code examples for the
>>> catalog manager part. In the meantime you can take a look at the WIP here:
>>>
>>> https://github.com/dvntucker/geotools/tree/im_api_refactor
>>>
>>> The biggest thing to note is that:
>>>
>>> - A number of methods in CatalogManager have been changed from static to
>>> non-static
>>> - CatalogManager has been added as a field to the
>>> ImageMosaicConfigHandler
>>> - Logic that was in ImageMosaicConfigHandler for checking ColorModel and
>>> CRS has been moved to CatalogManager (see the second part of the proposal)
>>>
>>> Cheers
>>>
>>> On Wed, Jun 8, 2016 at 1:08 PM, Jody Garnett 
>>> wrote:

 Thanks for email feedback Andrea, Daniele. Had a good chat with Simone
 

 Going to try and break this proposal into three small proposals on 1)
 index generation 2) dynamic processing 3) harvest flexibility.

 Some of the functionality (like data prep) is indeed already covered. I
 was not aware, for example, that geoserver importer could be used add
 granules to an existing mosaic (that is not functionality that can be done
 for vector import).

 --
 Jody Garnett

 On 5 April 2016 at 16:44, Devon Tucker  wrote:
>
> Hi all,
>
> Just noticed that there wasn't a dedicated email thread for this

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

Jody I'm not sure what a good name for the class is. Its chief
responsibilities (before this proposal) were collecting attributes from the
incoming granule, creating its new feature and updating the GranuleStore,
creating the index schema, and instantiating the GranuleCatalog from the
reader configuration. Now that I type all that out I don't think
CatalogManager is that bad a name?

On Tue, Jun 14, 2016 at 10:42 AM, Jody Garnett 
wrote:

> I note the prior proposal had CatalogManager renamed to
> GranuleCatalogManager is that still appropriate?
>
> Will you be removing the original proposal then?
>
> --
> Jody Garnett
>
> On 9 June 2016 at 16:39, Devon Tucker  wrote:
>
>> Thanks for all the feedback. Like Jody said, we're probably going to
>> break the proposal apart a bit. I've started pulling apart the
>> index/catalog management into its own proposal:
>>
>>
>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-Index-Catalog-management-for-improved-extensibility
>>
>> Still a work in progress. Will work on adding code examples for the
>> catalog manager part. In the meantime you can take a look at the WIP here:
>>
>> https://github.com/dvntucker/geotools/tree/im_api_refactor
>>
>> The biggest thing to note is that:
>>
>> - A number of methods in CatalogManager have been changed from static to
>> non-static
>> - CatalogManager has been added as a field to the ImageMosaicConfigHandler
>> - Logic that was in ImageMosaicConfigHandler for checking ColorModel and
>> CRS has been moved to CatalogManager (see the second part of the proposal)
>>
>> Cheers
>>
>> On Wed, Jun 8, 2016 at 1:08 PM, Jody Garnett 
>> wrote:
>>
>>> Thanks for email feedback Andrea, Daniele. Had a good chat with Simone
>>> 
>>>
>>> Going to try and break this proposal into three small proposals on 1)
>>> index generation 2) dynamic processing 3) harvest flexibility.
>>>
>>> Some of the functionality (like data prep) is indeed already covered. I
>>> was not aware, for example, that geoserver importer could be used add
>>> granules to an *existing* mosaic (that is not functionality that can be
>>> done for vector import).
>>>
>>> --
>>> Jody Garnett
>>>
>>> On 5 April 2016 at 16:44, Devon Tucker  wrote:
>>>
 Hi all,

 Just noticed that there wasn't a dedicated email thread for this
 proposal and figured I'd kick one off since I've also been involved in it.


 https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility

 I think the description gives a pretty good rationale for what we want
 to accomplish with this proposal and I'd like to solicit any discussion,
 advice, feedback, etc. from everyone.

 Cheers,
 Devon


 --

 ___
 GeoTools-Devel mailing list
 GeoTools-Devel@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/geotools-devel


>>>
>>
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

I note the prior proposal had CatalogManager renamed to
GranuleCatalogManager is that still appropriate?

Will you be removing the original proposal then?

--
Jody Garnett

On 9 June 2016 at 16:39, Devon Tucker  wrote:

> Thanks for all the feedback. Like Jody said, we're probably going to break
> the proposal apart a bit. I've started pulling apart the index/catalog
> management into its own proposal:
>
>
> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-Index-Catalog-management-for-improved-extensibility
>
> Still a work in progress. Will work on adding code examples for the
> catalog manager part. In the meantime you can take a look at the WIP here:
>
> https://github.com/dvntucker/geotools/tree/im_api_refactor
>
> The biggest thing to note is that:
>
> - A number of methods in CatalogManager have been changed from static to
> non-static
> - CatalogManager has been added as a field to the ImageMosaicConfigHandler
> - Logic that was in ImageMosaicConfigHandler for checking ColorModel and
> CRS has been moved to CatalogManager (see the second part of the proposal)
>
> Cheers
>
> On Wed, Jun 8, 2016 at 1:08 PM, Jody Garnett 
> wrote:
>
>> Thanks for email feedback Andrea, Daniele. Had a good chat with Simone
>> 
>>
>> Going to try and break this proposal into three small proposals on 1)
>> index generation 2) dynamic processing 3) harvest flexibility.
>>
>> Some of the functionality (like data prep) is indeed already covered. I
>> was not aware, for example, that geoserver importer could be used add
>> granules to an *existing* mosaic (that is not functionality that can be
>> done for vector import).
>>
>> --
>> Jody Garnett
>>
>> On 5 April 2016 at 16:44, Devon Tucker  wrote:
>>
>>> Hi all,
>>>
>>> Just noticed that there wasn't a dedicated email thread for this
>>> proposal and figured I'd kick one off since I've also been involved in it.
>>>
>>>
>>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>>
>>> I think the description gives a pretty good rationale for what we want
>>> to accomplish with this proposal and I'd like to solicit any discussion,
>>> advice, feedback, etc. from everyone.
>>>
>>> Cheers,
>>> Devon
>>>
>>>
>>> --
>>>
>>> ___
>>> GeoTools-Devel mailing list
>>> GeoTools-Devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>>
>>>
>>
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421=/41014381___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

Thanks for all the feedback. Like Jody said, we're probably going to break
the proposal apart a bit. I've started pulling apart the index/catalog
management into its own proposal:

https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-Index-Catalog-management-for-improved-extensibility

Still a work in progress. Will work on adding code examples for the catalog
manager part. In the meantime you can take a look at the WIP here:

https://github.com/dvntucker/geotools/tree/im_api_refactor

The biggest thing to note is that:

- A number of methods in CatalogManager have been changed from static to
non-static
- CatalogManager has been added as a field to the ImageMosaicConfigHandler
- Logic that was in ImageMosaicConfigHandler for checking ColorModel and
CRS has been moved to CatalogManager (see the second part of the proposal)

Cheers

On Wed, Jun 8, 2016 at 1:08 PM, Jody Garnett  wrote:

> Thanks for email feedback Andrea, Daniele. Had a good chat with Simone 
>
> Going to try and break this proposal into three small proposals on 1)
> index generation 2) dynamic processing 3) harvest flexibility.
>
> Some of the functionality (like data prep) is indeed already covered. I
> was not aware, for example, that geoserver importer could be used add
> granules to an *existing* mosaic (that is not functionality that can be
> done for vector import).
>
> --
> Jody Garnett
>
> On 5 April 2016 at 16:44, Devon Tucker  wrote:
>
>> Hi all,
>>
>> Just noticed that there wasn't a dedicated email thread for this proposal
>> and figured I'd kick one off since I've also been involved in it.
>>
>>
>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>
>> I think the description gives a pretty good rationale for what we want to
>> accomplish with this proposal and I'd like to solicit any discussion,
>> advice, feedback, etc. from everyone.
>>
>> Cheers,
>> Devon
>>
>>
>> --
>>
>> ___
>> GeoTools-Devel mailing list
>> GeoTools-Devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>
>>
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

Thanks for email feedback Andrea, Daniele. Had a good chat with Simone 

Going to try and break this proposal into three small proposals on 1) index
generation 2) dynamic processing 3) harvest flexibility.

Some of the functionality (like data prep) is indeed already covered. I was
not aware, for example, that geoserver importer could be used add granules
to an *existing* mosaic (that is not functionality that can be done for
vector import).

--
Jody Garnett

On 5 April 2016 at 16:44, Devon Tucker  wrote:

> Hi all,
>
> Just noticed that there wasn't a dedicated email thread for this proposal
> and figured I'd kick one off since I've also been involved in it.
>
>
> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>
> I think the description gives a pretty good rationale for what we want to
> accomplish with this proposal and I'd like to solicit any discussion,
> advice, feedback, etc. from everyone.
>
> Cheers,
> Devon
>
>
> --
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

Properly collectors are still used, think we are getting a small echo
chamber of uncertainty here. This is a refactor to use existing
infrastructure such as property accessors more effectively.

Importer is not quite the right place as additional granules can be added
over time via rest API right? And this is a GeoTools proposal 

On Tue, Jun 7, 2016 at 9:30 AM Daniele Romagnoli <
daniele.romagn...@geo-solutions.it> wrote:

> Hi,
> Providing some additional feedbacks on my side, too.
>
>
> On Mon, Jun 6, 2016 at 6:34 PM, Andrea Aime 
> wrote:
>
>> On Fri, Jun 3, 2016 at 1:19 AM, Devon Tucker 
>> wrote:
>>
>>> Here are a few things that might help understanding, section by section:

- *Description*: this is fine, makes sense, no questions (one
thing, the recursive file loading during indexing is already the default
behavior as far as I know)
- *Index Generation: *quite a bit lost here, how are
MosaicIndexConfiguration and GranuleCatalogManager going to be used by 
 the
existing classes? Or are they replacing it? I'd need to understand how 
 the
existing code is going to get reshuffled into a generic machiner plus
default implementations of the two above objects? Also, can you provide 
 an
example of an alternate implementation (being the refactor targeted to
extensibility)
- *Harvesting: *same as above, how is the code going to be moved
around, and examples of alternate implementations


>>> I hope to clarify these parts soon. To be honest, these parts were done
>>> earlier in the project and we've zig-zagged multiple times since then, so
>>> even I need to revisit to solidify and clarify this section. To give some
>>> perspective on the original motivations behind this part, at the time we
>>> were:
>>>
>>> - Performing some pre-processing on rasters before indexing them (adding
>>> overlays, creating alpha bands, generating elevation outlier footprints)
>>>
>>
>> Image mosaic looks like the wrong place to do this, the importer would be
>> much better (and already does some of those). See below for more
>> observations regarding this.
>>
> I would agree with Andrea's feedbacks on the pre-processing topic.
>
>
>>
>>
>>> - Collecting custom properties from rasters during indexing (resolution,
>>> date stamps, various tiff metadata, sample date, etc.)
>>>
>> Where this information will be stored? In the mentioned
> GranuleDescriptor's properties map? Shouldn't that achieved by
> PropertyCollector?
>
>>
>> This is done by property collectors, why not just extend that?
>>
>>
>>> - Looking at harvesting granules on demand either via. rest or the
>>> geoserver UI.
>>>
>>
>> The REST api already does that. There is no UI for it though.
>>
> Some more information about REST support for harvesting is available here
> with some examples:
>
> http://docs.geoserver.org/stable/en/user/rest/examples/curl.html#uploading-and-modifying-a-image-mosaic
>
>
>>
>>> - Exposing more of this configuration through a custom store and layer
>>> config in GeoServer.
>>>
>>
>> This can be arranged too, not sure it needs a refactor for that.
>>
>>
>>> - Index generation code is very isolated as it stands. It's tough to
>>> programmatically configure index generation.
>>>
>>
>> Ok, this one makes sense... the others do not seem to actually require a
>> refactor though.
>>
>>
>>>
>>>

- *Delegate coverage acceptance/rejection to a predicate object*:
makes sense I guess, so the reason to have this plugable is because you
might roll a collector that has less limitations than the default ones?

 Exactly.
>>>
>>
>> Ok but... why not keep the improved collectors in the mosaic instead?
>> They would benefit every user there (instead of just the gt-dem users)
>>
>>
>>>
- *Pre-process Granule Footprint Before Indexing*: makes sense,
thinking out loud is the footprint the only thing that needs 
 reprocessing?

 No, it's not really the only thing that need reprocessing. In fact
>>> PropertyCollectors have a tangentially related responsibility (and are used
>>> in roughly the same spot). Maybe this functionality could be combined
>>> somehow, and make monolithic some sort of monolithic visitor which would be
>>> responsible for:
>>>
>>> - Collecting properties from granules
>>> - Pre-processing the granule footprint if needed
>>> - Populating the index feature
>>>
>>> And so on. I'll have to think about this one a bit more, but as the
>>> proposal stands at least FootprintProcessor has a very clear responsibility.
>>>
>>
>>
>
>> See above about the usage of the importer. The indexing happens during
>> layer configuration in the UI, expensive operations are best done during
>> pre-processing.
>> Normally what you are describing here is done as part of a "ingest"
>> machinery, which is a off-line ETL of sorts 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Daniele Romagnoli]

Hi,
Providing some additional feedbacks on my side, too.


On Mon, Jun 6, 2016 at 6:34 PM, Andrea Aime 
wrote:

> On Fri, Jun 3, 2016 at 1:19 AM, Devon Tucker 
> wrote:
>
>> Here are a few things that might help understanding, section by section:
>>>
>>>- *Description*: this is fine, makes sense, no questions (one thing,
>>>the recursive file loading during indexing is already the default 
>>> behavior
>>>as far as I know)
>>>- *Index Generation: *quite a bit lost here, how are
>>>MosaicIndexConfiguration and GranuleCatalogManager going to be used by 
>>> the
>>>existing classes? Or are they replacing it? I'd need to understand how 
>>> the
>>>existing code is going to get reshuffled into a generic machiner plus
>>>default implementations of the two above objects? Also, can you provide 
>>> an
>>>example of an alternate implementation (being the refactor targeted to
>>>extensibility)
>>>- *Harvesting: *same as above, how is the code going to be moved
>>>around, and examples of alternate implementations
>>>
>>>
>> I hope to clarify these parts soon. To be honest, these parts were done
>> earlier in the project and we've zig-zagged multiple times since then, so
>> even I need to revisit to solidify and clarify this section. To give some
>> perspective on the original motivations behind this part, at the time we
>> were:
>>
>> - Performing some pre-processing on rasters before indexing them (adding
>> overlays, creating alpha bands, generating elevation outlier footprints)
>>
>
> Image mosaic looks like the wrong place to do this, the importer would be
> much better (and already does some of those). See below for more
> observations regarding this.
>
I would agree with Andrea's feedbacks on the pre-processing topic.


>
>
>> - Collecting custom properties from rasters during indexing (resolution,
>> date stamps, various tiff metadata, sample date, etc.)
>>
> Where this information will be stored? In the mentioned
GranuleDescriptor's properties map? Shouldn't that achieved by
PropertyCollector?

>
> This is done by property collectors, why not just extend that?
>
>
>> - Looking at harvesting granules on demand either via. rest or the
>> geoserver UI.
>>
>
> The REST api already does that. There is no UI for it though.
>
Some more information about REST support for harvesting is available here
with some examples:
http://docs.geoserver.org/stable/en/user/rest/examples/curl.html#uploading-and-modifying-a-image-mosaic


>
>> - Exposing more of this configuration through a custom store and layer
>> config in GeoServer.
>>
>
> This can be arranged too, not sure it needs a refactor for that.
>
>
>> - Index generation code is very isolated as it stands. It's tough to
>> programmatically configure index generation.
>>
>
> Ok, this one makes sense... the others do not seem to actually require a
> refactor though.
>
>
>>
>>
>>>
>>>- *Delegate coverage acceptance/rejection to a predicate object*:
>>>makes sense I guess, so the reason to have this plugable is because you
>>>might roll a collector that has less limitations than the default ones?
>>>
>>> Exactly.
>>
>
> Ok but... why not keep the improved collectors in the mosaic instead? They
> would benefit every user there (instead of just the gt-dem users)
>
>
>>
>>>- *Pre-process Granule Footprint Before Indexing*: makes sense,
>>>thinking out loud is the footprint the only thing that needs 
>>> reprocessing?
>>>
>>> No, it's not really the only thing that need reprocessing. In fact
>> PropertyCollectors have a tangentially related responsibility (and are used
>> in roughly the same spot). Maybe this functionality could be combined
>> somehow, and make monolithic some sort of monolithic visitor which would be
>> responsible for:
>>
>> - Collecting properties from granules
>> - Pre-processing the granule footprint if needed
>> - Populating the index feature
>>
>> And so on. I'll have to think about this one a bit more, but as the
>> proposal stands at least FootprintProcessor has a very clear responsibility.
>>
>
>

> See above about the usage of the importer. The indexing happens during
> layer configuration in the UI, expensive operations are best done during
> pre-processing.
> Normally what you are describing here is done as part of a "ingest"
> machinery, which is a off-line ETL of sorts that prepares the data for
> publication, but
> before it actually reaches GeoServer. The importer is one example of
> those, I've seen several others, all of them were asynch and offline.
>

I agree with this comment too.

>
>
>>
>>>- *Generalize Mosaicking per GranuleCollector and Update
>>>GranuleCollector to a tree-like hierarchy*: an example would be
>>>useful
>>>
>>> I can do an example for this. The motivation here is to delegate the
>> actual mosaicking to an object which may internally be delegating to other
>> mosaicking objects. For 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Andrea Aime]

On Fri, Jun 3, 2016 at 1:19 AM, Devon Tucker  wrote:

> Here are a few things that might help understanding, section by section:
>>
>>- *Description*: this is fine, makes sense, no questions (one thing,
>>the recursive file loading during indexing is already the default behavior
>>as far as I know)
>>- *Index Generation: *quite a bit lost here, how are
>>MosaicIndexConfiguration and GranuleCatalogManager going to be used by the
>>existing classes? Or are they replacing it? I'd need to understand how the
>>existing code is going to get reshuffled into a generic machiner plus
>>default implementations of the two above objects? Also, can you provide an
>>example of an alternate implementation (being the refactor targeted to
>>extensibility)
>>- *Harvesting: *same as above, how is the code going to be moved
>>around, and examples of alternate implementations
>>
>>
> I hope to clarify these parts soon. To be honest, these parts were done
> earlier in the project and we've zig-zagged multiple times since then, so
> even I need to revisit to solidify and clarify this section. To give some
> perspective on the original motivations behind this part, at the time we
> were:
>
> - Performing some pre-processing on rasters before indexing them (adding
> overlays, creating alpha bands, generating elevation outlier footprints)
>

Image mosaic looks like the wrong place to do this, the importer would be
much better (and already does some of those). See below for more
observations regarding this.


> - Collecting custom properties from rasters during indexing (resolution,
> date stamps, various tiff metadata, sample date, etc.)
>

This is done by property collectors, why not just extend that?


> - Looking at harvesting granules on demand either via. rest or the
> geoserver UI.
>

The REST api already does that. There is no UI for it though.


> - Exposing more of this configuration through a custom store and layer
> config in GeoServer.
>

This can be arranged too, not sure it needs a refactor for that.


> - Index generation code is very isolated as it stands. It's tough to
> programmatically configure index generation.
>

Ok, this one makes sense... the others do not seem to actually require a
refactor though.


>
>
>>
>>- *Delegate coverage acceptance/rejection to a predicate object*:
>>makes sense I guess, so the reason to have this plugable is because you
>>might roll a collector that has less limitations than the default ones?
>>
>> Exactly.
>

Ok but... why not keep the improved collectors in the mosaic instead? They
would benefit every user there (instead of just the gt-dem users)


>
>>- *Pre-process Granule Footprint Before Indexing*: makes sense,
>>thinking out loud is the footprint the only thing that needs reprocessing?
>>
>> No, it's not really the only thing that need reprocessing. In fact
> PropertyCollectors have a tangentially related responsibility (and are used
> in roughly the same spot). Maybe this functionality could be combined
> somehow, and make monolithic some sort of monolithic visitor which would be
> responsible for:
>
> - Collecting properties from granules
> - Pre-processing the granule footprint if needed
> - Populating the index feature
>
> And so on. I'll have to think about this one a bit more, but as the
> proposal stands at least FootprintProcessor has a very clear responsibility.
>

See above about the usage of the importer. The indexing happens during
layer configuration in the UI, expensive operations are best done during
pre-processing.
Normally what you are describing here is done as part of a "ingest"
machinery, which is a off-line ETL of sorts that prepares the data for
publication, but
before it actually reaches GeoServer. The importer is one example of those,
I've seen several others, all of them were asynch and offline.


>
>>- *Generalize Mosaicking per GranuleCollector and Update
>>GranuleCollector to a tree-like hierarchy*: an example would be useful
>>
>> I can do an example for this. The motivation here is to delegate the
> actual mosaicking to an object which may internally be delegating to other
> mosaicking objects. For example, the default implementation would have the
> logic for mosaicking per-resolution first, then resampling and mosaicking
> those results.
>

+1


>
>>- *Enhance the GranuleDescriptor and GranuleCatalogVisitor
>>interfaces: *same as above, examples of these "arbitrary properties"
>>to be used would be useful
>>
>> I will provide examples for this as well. In our RnD it was necessary to
> set the CRS on each granule descriptor.
>

Ok... but wouldn't it be better to keep the CRS information as an attribute
in the index instead?

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

Working on it with me, and we expect some review.
On Thu, Jun 2, 2016 at 4:52 PM Ben Caradoc-Davies  wrote:

> Devon,
>
> are you working on this by yourself or with others? With Git, you can
> make your own branch in your own fork of the repository, then use a
> GitHub pull request to ask for review and merge. Justin wrote our
> excellent Working with Git guide:
> http://docs.geotools.org/latest/developer/procedures/git.html
>
> A branch in the main repo can be useful if you are collaborating on this
> with several others, but most work in this project is done on feature
> branches in developer forks and merged into master via GitHub pull
> requests. I know that other Git platforms encourage centralised
> workflows based on branches in a shared repo; I do not know your
> background with Git. In this project I have mostly seen decentralised
> workflow based on developer forks. Git and GitHub are very flexible and
> you can also make pull requests to other developers' forks.
>
> Commit access is on a per-organisation basis (i.e. all of GeoTools); we
> cannot grant commit access to a single module. We rely on peer review
> and accountability. Even people with commit access often make pull
> requests to enable peer review, especially for core code. Review and
> debugging is highly parallelisable.
>
> Kind regards,
> Ben.
>
>
> On 03/06/16 11:25, Devon Tucker wrote:
> > Also, would it be possible to get commit access to the gt-dem plugin as
> > well as a branch created for the ImageMosaic API work?
> >
> > Thanks,
> > Devon
>
> --
> Ben Caradoc-Davies 
> Director
> Transient Software Limited 
> New Zealand
>
>
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and
> traffic
> patterns at an interface-level. Reveals which users, apps, and protocols
> are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
-- 
--
Jody Garnett
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Ben Caradoc-Davies]

Devon,

are you working on this by yourself or with others? With Git, you can 
make your own branch in your own fork of the repository, then use a 
GitHub pull request to ask for review and merge. Justin wrote our 
excellent Working with Git guide:
http://docs.geotools.org/latest/developer/procedures/git.html

A branch in the main repo can be useful if you are collaborating on this 
with several others, but most work in this project is done on feature 
branches in developer forks and merged into master via GitHub pull 
requests. I know that other Git platforms encourage centralised 
workflows based on branches in a shared repo; I do not know your 
background with Git. In this project I have mostly seen decentralised 
workflow based on developer forks. Git and GitHub are very flexible and 
you can also make pull requests to other developers' forks.

Commit access is on a per-organisation basis (i.e. all of GeoTools); we 
cannot grant commit access to a single module. We rely on peer review 
and accountability. Even people with commit access often make pull 
requests to enable peer review, especially for core code. Review and 
debugging is highly parallelisable.

Kind regards,
Ben.


On 03/06/16 11:25, Devon Tucker wrote:
> Also, would it be possible to get commit access to the gt-dem plugin as
> well as a branch created for the ImageMosaic API work?
>
> Thanks,
> Devon

-- 
Ben Caradoc-Davies 
Director
Transient Software Limited 
New Zealand

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

Also, would it be possible to get commit access to the gt-dem plugin as
well as a branch created for the ImageMosaic API work?

Thanks,
Devon

On Thu, Jun 2, 2016 at 4:19 PM, Devon Tucker  wrote:

> Hi Andrea,
>
> Thanks for the questions.
>
> On Wed, Jun 1, 2016 at 11:02 AM, Andrea Aime  > wrote:
>
>> Hi Jody,
>> at present I cannot vote on the proposal because I have troubles getting
>> a grip on it.
>>
>> Here are a few things that might help understanding, section by section:
>>
>>- *Description*: this is fine, makes sense, no questions (one thing,
>>the recursive file loading during indexing is already the default behavior
>>as far as I know)
>>- *Index Generation: *quite a bit lost here, how are
>>MosaicIndexConfiguration and GranuleCatalogManager going to be used by the
>>existing classes? Or are they replacing it? I'd need to understand how the
>>existing code is going to get reshuffled into a generic machiner plus
>>default implementations of the two above objects? Also, can you provide an
>>example of an alternate implementation (being the refactor targeted to
>>extensibility)
>>- *Harvesting: *same as above, how is the code going to be moved
>>around, and examples of alternate implementations
>>
>>
> I hope to clarify these parts soon. To be honest, these parts were done
> earlier in the project and we've zig-zagged multiple times since then, so
> even I need to revisit to solidify and clarify this section. To give some
> perspective on the original motivations behind this part, at the time we
> were:
>
> - Performing some pre-processing on rasters before indexing them (adding
> overlays, creating alpha bands, generating elevation outlier footprints)
> - Collecting custom properties from rasters during indexing (resolution,
> date stamps, various tiff metadata, sample date, etc.)
> - Looking at harvesting granules on demand either via. rest or the
> geoserver UI.
> - Exposing more of this configuration through a custom store and layer
> config in GeoServer.
> - Index generation code is very isolated as it stands. It's tough to
> programmatically configure index generation.
>
>
>>
>>- *Delegate coverage acceptance/rejection to a predicate object*:
>>makes sense I guess, so the reason to have this plugable is because you
>>might roll a collector that has less limitations than the default ones?
>>
>> Exactly.
>
>>
>>- *Pre-process Granule Footprint Before Indexing*: makes sense,
>>thinking out loud is the footprint the only thing that needs reprocessing?
>>
>> No, it's not really the only thing that need reprocessing. In fact
> PropertyCollectors have a tangentially related responsibility (and are used
> in roughly the same spot). Maybe this functionality could be combined
> somehow, and make monolithic some sort of monolithic visitor which would be
> responsible for:
>
> - Collecting properties from granules
> - Pre-processing the granule footprint if needed
> - Populating the index feature
>
> And so on. I'll have to think about this one a bit more, but as the
> proposal stands at least FootprintProcessor has a very clear responsibility.
>
>>
>>- *Generalize Mosaicking per GranuleCollector and Update
>>GranuleCollector to a tree-like hierarchy*: an example would be useful
>>
>> I can do an example for this. The motivation here is to delegate the
> actual mosaicking to an object which may internally be delegating to other
> mosaicking objects. For example, the default implementation would have the
> logic for mosaicking per-resolution first, then resampling and mosaicking
> those results.
>
>>
>>- *Enhance the GranuleDescriptor and GranuleCatalogVisitor
>>interfaces: *same as above, examples of these "arbitrary properties"
>>to be used would be useful
>>
>> I will provide examples for this as well. In our RnD it was necessary to
> set the CRS on each granule descriptor.
>
> Hope this clarifies a bit. I expect to be working on this the rest of the
> week and be able to answer remaining questions.
>
> Cheers
>
> Cheers
>> Andrea
>>
>>
>> On Wed, Jun 1, 2016 at 2:48 AM, Jody Garnett 
>> wrote:
>>
>>> Proposal is updated:
>>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>>
>>> I reworded the API change section with clear BEFORE/AFTER descriptions
>>> making it easier to follow. After this I am far more comfortable with the
>>> proposal as a whole. I have some specific class naming doubts (is
>>> GranuleCatalogManager really a manager) but they can be sorted out during
>>> the refactor.
>>>
>>> +1
>>>
>>> A couple feedbacks:

>>> - (done) go ahead and merge back to master, we can edit there as a group
 :)
 - (done) for the API change on MosaicIndexConfiguration - can you take
 the cometary out to some bullet points so we can see the proposed class in
 one go
 - (done) the tasks 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

Hi Andrea,

Thanks for the questions.

On Wed, Jun 1, 2016 at 11:02 AM, Andrea Aime 
wrote:

> Hi Jody,
> at present I cannot vote on the proposal because I have troubles getting a
> grip on it.
>
> Here are a few things that might help understanding, section by section:
>
>- *Description*: this is fine, makes sense, no questions (one thing,
>the recursive file loading during indexing is already the default behavior
>as far as I know)
>- *Index Generation: *quite a bit lost here, how are
>MosaicIndexConfiguration and GranuleCatalogManager going to be used by the
>existing classes? Or are they replacing it? I'd need to understand how the
>existing code is going to get reshuffled into a generic machiner plus
>default implementations of the two above objects? Also, can you provide an
>example of an alternate implementation (being the refactor targeted to
>extensibility)
>- *Harvesting: *same as above, how is the code going to be moved
>around, and examples of alternate implementations
>
>
I hope to clarify these parts soon. To be honest, these parts were done
earlier in the project and we've zig-zagged multiple times since then, so
even I need to revisit to solidify and clarify this section. To give some
perspective on the original motivations behind this part, at the time we
were:

- Performing some pre-processing on rasters before indexing them (adding
overlays, creating alpha bands, generating elevation outlier footprints)
- Collecting custom properties from rasters during indexing (resolution,
date stamps, various tiff metadata, sample date, etc.)
- Looking at harvesting granules on demand either via. rest or the
geoserver UI.
- Exposing more of this configuration through a custom store and layer
config in GeoServer.
- Index generation code is very isolated as it stands. It's tough to
programmatically configure index generation.


>
>- *Delegate coverage acceptance/rejection to a predicate object*:
>makes sense I guess, so the reason to have this plugable is because you
>might roll a collector that has less limitations than the default ones?
>
> Exactly.

>
>- *Pre-process Granule Footprint Before Indexing*: makes sense,
>thinking out loud is the footprint the only thing that needs reprocessing?
>
> No, it's not really the only thing that need reprocessing. In fact
PropertyCollectors have a tangentially related responsibility (and are used
in roughly the same spot). Maybe this functionality could be combined
somehow, and make monolithic some sort of monolithic visitor which would be
responsible for:

- Collecting properties from granules
- Pre-processing the granule footprint if needed
- Populating the index feature

And so on. I'll have to think about this one a bit more, but as the
proposal stands at least FootprintProcessor has a very clear responsibility.

>
>- *Generalize Mosaicking per GranuleCollector and Update
>GranuleCollector to a tree-like hierarchy*: an example would be useful
>
> I can do an example for this. The motivation here is to delegate the
actual mosaicking to an object which may internally be delegating to other
mosaicking objects. For example, the default implementation would have the
logic for mosaicking per-resolution first, then resampling and mosaicking
those results.

>
>- *Enhance the GranuleDescriptor and GranuleCatalogVisitor interfaces:
>*same as above, examples of these "arbitrary properties" to be used
>would be useful
>
> I will provide examples for this as well. In our RnD it was necessary to
set the CRS on each granule descriptor.

Hope this clarifies a bit. I expect to be working on this the rest of the
week and be able to answer remaining questions.

Cheers

Cheers
> Andrea
>
>
> On Wed, Jun 1, 2016 at 2:48 AM, Jody Garnett 
> wrote:
>
>> Proposal is updated:
>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>
>> I reworded the API change section with clear BEFORE/AFTER descriptions
>> making it easier to follow. After this I am far more comfortable with the
>> proposal as a whole. I have some specific class naming doubts (is
>> GranuleCatalogManager really a manager) but they can be sorted out during
>> the refactor.
>>
>> +1
>>
>> A couple feedbacks:
>>>
>> - (done) go ahead and merge back to master, we can edit there as a group
>>> :)
>>> - (done) for the API change on MosaicIndexConfiguration - can you take
>>> the cometary out to some bullet points so we can see the proposed class in
>>> one go
>>> - (done) the tasks section seems incomplete, we mostly use this to check
>>> that you have enough resources/time to get the work done
>>> - loadGranuleCatalogFromDataStore seems a bit odd to make, taking a
>>> properties file (I guess of connection parameters) rather than a DataStore?
>>> We have the Repository API that meets this need for app-schema and
>>> pregeneralized datastore (I do not 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

I am not the original author, although I did revise yesterday for each api
change to have a motivation/before/after story.

Additional comments inline:

On 1 June 2016 at 11:02, Andrea Aime  wrote:

>
>- *Description*: this is fine, makes sense, no questions (one thing,
>the recursive file loading during indexing is already the default behavior
>as far as I know)
>
> Okay, not sure I am that close to it.

>
>- *Index Generation: *quite a bit lost here, how are
>MosaicIndexConfiguration and GranuleCatalogManager going to be used by the
>existing classes? Or are they replacing it? I'd need to understand how the
>existing code is going to get reshuffled into a generic machiner plus
>default implementations of the two above objects? Also, can you provide an
>example of an alternate implementation (being the refactor targeted to
>extensibility)
>
> They will be used by the existing classes, the logic has been isolated
into MosaicIndexConfiguration rather than dispersed across the codebase.
CatalogMangaer is an example of how this is done (it was also renamed to
GranuleCatalogManager because I found the name rather generic).

>
>- *Harvesting: *same as above, how is the code going to be moved
>around, and examples of alternate implementations
>
>  We may be able to pull together an example of an alternate implementation
from the RnD fork, but the RnD fork has this logic scattered across the
codebase so it would not be definitive.

>
>- *Delegate coverage acceptance/rejection to a predicate object*:
>makes sense I guess, so the reason to have this plugable is because you
>might roll a collector that has less limitations than the default ones?
>
> Exactly.

>
>- *Pre-process Granule Footprint Before Indexing*: makes sense,
>thinking out loud is the footprint the only thing that needs reprocessing?
>
> Only reprocessing that was identified in RnD fork, however I suspect that
properties collectors may of been used to reprocess (but that is just an
educated guess based on the API rather than proof).

>
>- *Generalize Mosaicking per GranuleCollector and Update
>GranuleCollector to a tree-like hierarchy*: an example would be useful
>
>
>- *Enhance the GranuleDescriptor and GranuleCatalogVisitor interfaces:
>*same as above, examples of these "arbitrary properties" to be used
>would be useful
>
> data collection metadata from the geotiff header, file creation /
modification dates, resolution ... or assumptions of sensor (and thus
accuracy) based on file naming convention).

Thanks for feedback.
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Andrea Aime]

Hi Jody,
at present I cannot vote on the proposal because I have troubles getting a
grip on it.

Here are a few things that might help understanding, section by section:

   - *Description*: this is fine, makes sense, no questions (one thing, the
   recursive file loading during indexing is already the default behavior as
   far as I know)
   - *Index Generation: *quite a bit lost here, how are
   MosaicIndexConfiguration and GranuleCatalogManager going to be used by the
   existing classes? Or are they replacing it? I'd need to understand how the
   existing code is going to get reshuffled into a generic machiner plus
   default implementations of the two above objects? Also, can you provide an
   example of an alternate implementation (being the refactor targeted to
   extensibility)
   - *Harvesting: *same as above, how is the code going to be moved around,
   and examples of alternate implementations
   - *Delegate coverage acceptance/rejection to a predicate object*: makes
   sense I guess, so the reason to have this plugable is because you might
   roll a collector that has less limitations than the default ones?
   - *Pre-process Granule Footprint Before Indexing*: makes sense, thinking
   out loud is the footprint the only thing that needs reprocessing?
   - *Generalize Mosaicking per GranuleCollector and Update
   GranuleCollector to a tree-like hierarchy*: an example would be useful
   - *Enhance the GranuleDescriptor and GranuleCatalogVisitor interfaces: *same
   as above, examples of these "arbitrary properties" to be used would be
   useful

Cheers
Andrea


On Wed, Jun 1, 2016 at 2:48 AM, Jody Garnett  wrote:

> Proposal is updated:
> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>
> I reworded the API change section with clear BEFORE/AFTER descriptions
> making it easier to follow. After this I am far more comfortable with the
> proposal as a whole. I have some specific class naming doubts (is
> GranuleCatalogManager really a manager) but they can be sorted out during
> the refactor.
>
> +1
>
> A couple feedbacks:
>>
> - (done) go ahead and merge back to master, we can edit there as a group :)
>> - (done) for the API change on MosaicIndexConfiguration - can you take
>> the cometary out to some bullet points so we can see the proposed class in
>> one go
>> - (done) the tasks section seems incomplete, we mostly use this to check
>> that you have enough resources/time to get the work done
>> - loadGranuleCatalogFromDataStore seems a bit odd to make, taking a
>> properties file (I guess of connection parameters) rather than a DataStore?
>> We have the Repository API that meets this need for app-schema and
>> pregeneralized datastore (I do not think you will have scope to address
>> this one)
>> - MosaicIndexConfiguration <-- is this really a configuration? It looks
>> to be more of a strategy
>> - (done) I am not familiar with this codebase so a diagram (say from
>> objectaid ) could help
>> - (done) Consider moving your before and after code examples, and a
>> diagram up to the description section :)
>> --
>> Jody Garnett
>>
>
>
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and
> traffic
> patterns at an interface-level. Reveals which users, apps, and protocols
> are
> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
> J-Flow, sFlow and other flows. Make informed decisions using capacity
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

Proposal is updated:
https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility

I reworded the API change section with clear BEFORE/AFTER descriptions
making it easier to follow. After this I am far more comfortable with the
proposal as a whole. I have some specific class naming doubts (is
GranuleCatalogManager really a manager) but they can be sorted out during
the refactor.

+1

A couple feedbacks:
>
- (done) go ahead and merge back to master, we can edit there as a group :)
> - (done) for the API change on MosaicIndexConfiguration - can you take the
> cometary out to some bullet points so we can see the proposed class in one
> go
> - (done) the tasks section seems incomplete, we mostly use this to check
> that you have enough resources/time to get the work done
> - loadGranuleCatalogFromDataStore seems a bit odd to make, taking a
> properties file (I guess of connection parameters) rather than a DataStore?
> We have the Repository API that meets this need for app-schema and
> pregeneralized datastore (I do not think you will have scope to address
> this one)
> - MosaicIndexConfiguration <-- is this really a configuration? It looks to
> be more of a strategy
> - (done) I am not familiar with this codebase so a diagram (say from
> objectaid ) could help
> - (done) Consider moving your before and after code examples, and a
> diagram up to the description section :)
> --
> Jody Garnett
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

A couple feedbacks:

- go ahead and merge back to master, we can edit there as a group :)
- for the API change on MosaicIndexConfiguration - can you take the
cometary out to some bullet points so we can see the proposed class in one
go
- the tasks section seems incomplete, we mostly use this to check that you
have enough resources/time to get the work done
- loadGranuleCatalogFromDataStore seems a bit odd to make, taking a
properties file (I guess of connection parameters) rather than a DataStore?
We have the Repository API that meets this need for app-schema and
pregeneralized datastore (I do not think you will have scope to address
this one)
- MosaicIndexConfiguration <-- is this really a configuration? It looks to
be more of a strategy
- I am not familiar with this codebase so a diagram (say from objectaid )
could help
- Consider moving your before and after code examples, and a diagram up to
the description section :)



--
Jody Garnett

On 20 May 2016 at 16:21, Devon Tucker  wrote:

> Hi all,
>
> I've been working on fleshing this proposal out and adding a bit to it.
> See the conversation on on-the-fly reprojection in ImageMosaic. My updated
> API proposal is on my own Wiki for now, I can update the master one but I
> wanted to solicit early feedback. Please let me know if something doesn't
> make sense. Wanted to throw it out there earlier for feedback rather than
> later:
>
>
> https://github.com/dvntucker/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>
> Cheers,
> Devon
>
> On Fri, Apr 22, 2016 at 1:20 AM, Niels Charlier  wrote:
>
>> On 21-04-16 20:01, Jody Garnett wrote:
>>
>> Thanks for the clarification Niels, I was not aware that Prop was part of
>> the current API.
>>
>> Still we have some duplication here right? Should we try running the
>> property collectors, look at the results, and generate the schema
>> appropriate to store the results?
>>
>>
>> Jody, have a look at this:
>> http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_indexer.html#imagemosaic-xml-indexer-example
>> You see that in the current index configuration file, there is a separate
>> section for the schemas and the property collectors. Both sections provide
>> a completely different kind of information.
>>
>> I do not think that there is duplication. You cannot run property
>> collectors if you do not have a schema yet. The property collectors are run
>> per feature (granule). The schema is made before you can even create
>> features. The schema is defined per coverage inside the store and can
>> differ between them. The property collectors are unaware of the coverages,
>> and they may operate for multiple. Once the schema has been created for a
>> coverage, the features are created and then values might be needed for each
>> feature. This is the time that (for some properties) the appropriate
>> property collectors are looked up and called to fill in those values.
>> In summary:
>> the schema -> defines the metadata (table structure) of the index per
>> coverage
>> the property collectors -> defines how to grab the data for the index
>> (from the metadata of the granules)
>> two different functions altogether.
>>
>>
>>
>>
>> --
>> Jody Garnett
>>
>> On 21 April 2016 at 03:00, Niels Charlier  wrote:
>>
>>> On 17-04-16 04:17, Jody Garnett wrote:
>>>
 This proposal (and email thread) should be orthogonal to the RnD
 discussion on setting up a mosaic with multiple projections.

 This mosaic is focused on creating/managing the index used. One part I
 cannot wrap my head around is the workflow behind these two methods -
 createSchema(String name, CoordinateReferenceSystem crs) which creates the
 FeatureType used for the index, and Collection
 getPropertiesCollectors().
 It seems these two methods need to be implemented together, a single
 PropertyCollector may collect more than once value on examining a raster. I
 am not sure how to line up these collected values with the created schema
 without more information.

>>>
>>> Just to be clear, the proposal doesn't change anything about
>>> PropertyCollectors work. This would simply continue to work as it is
>>> working right now, until somebody makes a proposal to change it to
>>> something else!
>>>
>>> These two things have a completely different function. The schema
>>> defines the structure of the index.
>>>
>>> Property Collectors are customisable mechanisms that provides (some of
>>> the) values of the index per granule, taking information from the file
>>> and/or the coverage metadata.
>>>
>>> The schema creation includes attributes that are not collected with a
>>> PropertyCollector (such as location), while one Propery Collector can
>>> potentially supply multiple attribute values.
>>>
>>>
>>>  We also have String getParameter(Prop prop) defined, but the class Prop
 is not included in the proposal.

>>>
>>> Prop already exists in the 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Niels Charlier]

Devon,

Great work, thanks for improving the explanation of my part. I'm glad to 
see the plan is moving forward.


Kind Regards
Niels

On 21-05-16 01:21, Devon Tucker wrote:

Hi all,

I've been working on fleshing this proposal out and adding a bit to 
it. See the conversation on on-the-fly reprojection in ImageMosaic. My 
updated API proposal is on my own Wiki for now, I can update the 
master one but I wanted to solicit early feedback. Please let me know 
if something doesn't make sense. Wanted to throw it out there earlier 
for feedback rather than later:


https://github.com/dvntucker/geotools/wiki/Refactor-ImageMosaic-for-extensibility

Cheers,
Devon

On Fri, Apr 22, 2016 at 1:20 AM, Niels Charlier > wrote:


On 21-04-16 20:01, Jody Garnett wrote:

Thanks for the clarification Niels, I was not aware that Prop was
part of the current API.

Still we have some duplication here right? Should we try running
the property collectors, look at the results, and generate the
schema appropriate to store the results?


Jody, have a look at this:

http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_indexer.html#imagemosaic-xml-indexer-example
You see that in the current index configuration file, there is a
separate section for the schemas and the property collectors. Both
sections provide a completely different kind of information.

I do not think that there is duplication. You cannot run property
collectors if you do not have a schema yet. The property
collectors are run per feature (granule). The schema is made
before you can even create features. The schema is defined per
coverage inside the store and can differ between them. The
property collectors are unaware of the coverages, and they may
operate for multiple. Once the schema has been created for a
coverage, the features are created and then values might be needed
for each feature. This is the time that (for some properties) the
appropriate property collectors are looked up and called to fill
in those values.
In summary:
the schema -> defines the metadata (table structure) of the index
per coverage
the property collectors -> defines how to grab the data for the
index (from the metadata of the granules)
two different functions altogether.





--
Jody Garnett

On 21 April 2016 at 03:00, Niels Charlier > wrote:

On 17-04-16 04:17, Jody Garnett wrote:

This proposal (and email thread) should be orthogonal to
the RnD discussion on setting up a mosaic with multiple
projections.

This mosaic is focused on creating/managing the index
used. One part I cannot wrap my head around is the
workflow behind these two methods - createSchema(String
name, CoordinateReferenceSystem crs) which creates the
FeatureType used for the index, and
Collection getPropertiesCollectors().
It seems these two methods need to be implemented
together, a single PropertyCollector may collect more
than once value on examining a raster. I am not sure how
to line up these collected values with the created schema
without more information.


Just to be clear, the proposal doesn't change anything about
PropertyCollectors work. This would simply continue to work
as it is working right now, until somebody makes a proposal
to change it to something else!

These two things have a completely different function. The
schema defines the structure of the index.

Property Collectors are customisable mechanisms that provides
(some of the) values of the index per granule, taking
information from the file and/or the coverage metadata.

The schema creation includes attributes that are not
collected with a PropertyCollector (such as location), while
one Propery Collector can potentially supply multiple
attribute values.


 We also have String getParameter(Prop prop) defined, but
the class Prop is not included in the proposal.


Prop already exists in the current API.


Suggestions:
- Use a list of PropertiesCollectors to provide order,
which can then be used to determine the Schema?
- Language consistency - raster / granule
- Language consistency - could we use AttributeDescriptor
rather than introduce a new thing called "Prop" ?


You do know that Props are configuration properties, nothing
to do with features?

- Are the class responsibilities clear:
GranuleCatalogManager, GranuleCatalog, GranuleStore,
MosaicHarvester, etc...

(It could also be that 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

Hi all,

I've been working on fleshing this proposal out and adding a bit to it. See
the conversation on on-the-fly reprojection in ImageMosaic. My updated API
proposal is on my own Wiki for now, I can update the master one but I
wanted to solicit early feedback. Please let me know if something doesn't
make sense. Wanted to throw it out there earlier for feedback rather than
later:

https://github.com/dvntucker/geotools/wiki/Refactor-ImageMosaic-for-extensibility

Cheers,
Devon

On Fri, Apr 22, 2016 at 1:20 AM, Niels Charlier  wrote:

> On 21-04-16 20:01, Jody Garnett wrote:
>
> Thanks for the clarification Niels, I was not aware that Prop was part of
> the current API.
>
> Still we have some duplication here right? Should we try running the
> property collectors, look at the results, and generate the schema
> appropriate to store the results?
>
>
> Jody, have a look at this:
> http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_indexer.html#imagemosaic-xml-indexer-example
> You see that in the current index configuration file, there is a separate
> section for the schemas and the property collectors. Both sections provide
> a completely different kind of information.
>
> I do not think that there is duplication. You cannot run property
> collectors if you do not have a schema yet. The property collectors are run
> per feature (granule). The schema is made before you can even create
> features. The schema is defined per coverage inside the store and can
> differ between them. The property collectors are unaware of the coverages,
> and they may operate for multiple. Once the schema has been created for a
> coverage, the features are created and then values might be needed for each
> feature. This is the time that (for some properties) the appropriate
> property collectors are looked up and called to fill in those values.
> In summary:
> the schema -> defines the metadata (table structure) of the index per
> coverage
> the property collectors -> defines how to grab the data for the index
> (from the metadata of the granules)
> two different functions altogether.
>
>
>
>
> --
> Jody Garnett
>
> On 21 April 2016 at 03:00, Niels Charlier  wrote:
>
>> On 17-04-16 04:17, Jody Garnett wrote:
>>
>>> This proposal (and email thread) should be orthogonal to the RnD
>>> discussion on setting up a mosaic with multiple projections.
>>>
>>> This mosaic is focused on creating/managing the index used. One part I
>>> cannot wrap my head around is the workflow behind these two methods -
>>> createSchema(String name, CoordinateReferenceSystem crs) which creates the
>>> FeatureType used for the index, and Collection
>>> getPropertiesCollectors().
>>> It seems these two methods need to be implemented together, a single
>>> PropertyCollector may collect more than once value on examining a raster. I
>>> am not sure how to line up these collected values with the created schema
>>> without more information.
>>>
>>
>> Just to be clear, the proposal doesn't change anything about
>> PropertyCollectors work. This would simply continue to work as it is
>> working right now, until somebody makes a proposal to change it to
>> something else!
>>
>> These two things have a completely different function. The schema defines
>> the structure of the index.
>>
>> Property Collectors are customisable mechanisms that provides (some of
>> the) values of the index per granule, taking information from the file
>> and/or the coverage metadata.
>>
>> The schema creation includes attributes that are not collected with a
>> PropertyCollector (such as location), while one Propery Collector can
>> potentially supply multiple attribute values.
>>
>>
>>  We also have String getParameter(Prop prop) defined, but the class Prop
>>> is not included in the proposal.
>>>
>>
>> Prop already exists in the current API.
>>
>>
>>> Suggestions:
>>> - Use a list of PropertiesCollectors to provide order, which can then be
>>> used to determine the Schema?
>>> - Language consistency - raster / granule
>>> - Language consistency - could we use AttributeDescriptor rather than
>>> introduce a new thing called "Prop" ?
>>>
>>
>> You do know that Props are configuration properties, nothing to do with
>> features?
>>
>> - Are the class responsibilities clear: GranuleCatalogManager,
>>> GranuleCatalog, GranuleStore, MosaicHarvester, etc...
>>>
>>> (It could also be that since I am unfamiliar with the internals being
>>> refactored that I am just struggling with the language - so if everyone
>>> else is okay ...)
>>>
>>>
>>> Regards
>> Niels
>>
>
>
>
>
> --
> Find and fix application performance issues faster with Applications
> Manager
> Applications Manager provides deep performance insights into multiple
> tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Niels Charlier]

On 21-04-16 20:01, Jody Garnett wrote:
Thanks for the clarification Niels, I was not aware that Prop was part 
of the current API.


Still we have some duplication here right? Should we try running the 
property collectors, look at the results, and generate the schema 
appropriate to store the results?


Jody, have a look at this: 
http://geoserver.geo-solutions.it/multidim/en/imagemosaic/mosaic_indexer.html#imagemosaic-xml-indexer-example
You see that in the current index configuration file, there is a 
separate section for the schemas and the property collectors. Both 
sections provide a completely different kind of information.


I do not think that there is duplication. You cannot run property 
collectors if you do not have a schema yet. The property collectors are 
run per feature (granule). The schema is made before you can even create 
features. The schema is defined per coverage inside the store and can 
differ between them. The property collectors are unaware of the 
coverages, and they may operate for multiple. Once the schema has been 
created for a coverage, the features are created and then values might 
be needed for each feature. This is the time that (for some properties) 
the appropriate property collectors are looked up and called to fill in 
those values.

In summary:
the schema -> defines the metadata (table structure) of the index per 
coverage
the property collectors -> defines how to grab the data for the index 
(from the metadata of the granules)

two different functions altogether.





--
Jody Garnett

On 21 April 2016 at 03:00, Niels Charlier > wrote:


On 17-04-16 04:17, Jody Garnett wrote:

This proposal (and email thread) should be orthogonal to the
RnD discussion on setting up a mosaic with multiple projections.

This mosaic is focused on creating/managing the index used.
One part I cannot wrap my head around is the workflow behind
these two methods - createSchema(String name,
CoordinateReferenceSystem crs) which creates the FeatureType
used for the index, and Collection
getPropertiesCollectors().
It seems these two methods need to be implemented together, a
single PropertyCollector may collect more than once value on
examining a raster. I am not sure how to line up these
collected values with the created schema without more information.


Just to be clear, the proposal doesn't change anything about
PropertyCollectors work. This would simply continue to work as it
is working right now, until somebody makes a proposal to change it
to something else!

These two things have a completely different function. The schema
defines the structure of the index.

Property Collectors are customisable mechanisms that provides
(some of the) values of the index per granule, taking information
from the file and/or the coverage metadata.

The schema creation includes attributes that are not collected
with a PropertyCollector (such as location), while one Propery
Collector can potentially supply multiple attribute values.


 We also have String getParameter(Prop prop) defined, but the
class Prop is not included in the proposal.


Prop already exists in the current API.


Suggestions:
- Use a list of PropertiesCollectors to provide order, which
can then be used to determine the Schema?
- Language consistency - raster / granule
- Language consistency - could we use AttributeDescriptor
rather than introduce a new thing called "Prop" ?


You do know that Props are configuration properties, nothing to do
with features?

- Are the class responsibilities clear: GranuleCatalogManager,
GranuleCatalog, GranuleStore, MosaicHarvester, etc...

(It could also be that since I am unfamiliar with the
internals being refactored that I am just struggling with the
language - so if everyone else is okay ...)


Regards
Niels




--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

Thanks for the clarification Niels, I was not aware that Prop was part of
the current API.

Still we have some duplication here right? Should we try running the
property collectors, look at the results, and generate the schema
appropriate to store the results?

--
Jody Garnett

On 21 April 2016 at 03:00, Niels Charlier  wrote:

> On 17-04-16 04:17, Jody Garnett wrote:
>
>> This proposal (and email thread) should be orthogonal to the RnD
>> discussion on setting up a mosaic with multiple projections.
>>
>> This mosaic is focused on creating/managing the index used. One part I
>> cannot wrap my head around is the workflow behind these two methods -
>> createSchema(String name, CoordinateReferenceSystem crs) which creates the
>> FeatureType used for the index, and Collection
>> getPropertiesCollectors().
>> It seems these two methods need to be implemented together, a single
>> PropertyCollector may collect more than once value on examining a raster. I
>> am not sure how to line up these collected values with the created schema
>> without more information.
>>
>
> Just to be clear, the proposal doesn't change anything about
> PropertyCollectors work. This would simply continue to work as it is
> working right now, until somebody makes a proposal to change it to
> something else!
>
> These two things have a completely different function. The schema defines
> the structure of the index.
>
> Property Collectors are customisable mechanisms that provides (some of
> the) values of the index per granule, taking information from the file
> and/or the coverage metadata.
>
> The schema creation includes attributes that are not collected with a
> PropertyCollector (such as location), while one Propery Collector can
> potentially supply multiple attribute values.
>
>
>  We also have String getParameter(Prop prop) defined, but the class Prop
>> is not included in the proposal.
>>
>
> Prop already exists in the current API.
>
>
>> Suggestions:
>> - Use a list of PropertiesCollectors to provide order, which can then be
>> used to determine the Schema?
>> - Language consistency - raster / granule
>> - Language consistency - could we use AttributeDescriptor rather than
>> introduce a new thing called "Prop" ?
>>
>
> You do know that Props are configuration properties, nothing to do with
> features?
>
> - Are the class responsibilities clear: GranuleCatalogManager,
>> GranuleCatalog, GranuleStore, MosaicHarvester, etc...
>>
>> (It could also be that since I am unfamiliar with the internals being
>> refactored that I am just struggling with the language - so if everyone
>> else is okay ...)
>>
>>
>> Regards
> Niels
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Niels Charlier]

On 17-04-16 04:17, Jody Garnett wrote:
> This proposal (and email thread) should be orthogonal to the RnD 
> discussion on setting up a mosaic with multiple projections.
>
> This mosaic is focused on creating/managing the index used. One part I 
> cannot wrap my head around is the workflow behind these two methods 
> - createSchema(String name, CoordinateReferenceSystem crs) which 
> creates the FeatureType used for the index, 
> and Collection getPropertiesCollectors().
> It seems these two methods need to be implemented together, a single 
> PropertyCollector may collect more than once value on examining a 
> raster. I am not sure how to line up these collected values with the 
> created schema without more information.

Just to be clear, the proposal doesn't change anything about 
PropertyCollectors work. This would simply continue to work as it is 
working right now, until somebody makes a proposal to change it to 
something else!

These two things have a completely different function. The schema 
defines the structure of the index.

Property Collectors are customisable mechanisms that provides (some of 
the) values of the index per granule, taking information from the file 
and/or the coverage metadata.

The schema creation includes attributes that are not collected with a 
PropertyCollector (such as location), while one Propery Collector can 
potentially supply multiple attribute values.


>  We also have String getParameter(Prop prop) defined, but the class 
> Prop is not included in the proposal.

Prop already exists in the current API.

>
> Suggestions:
> - Use a list of PropertiesCollectors to provide order, which can then 
> be used to determine the Schema?
> - Language consistency - raster / granule
> - Language consistency - could we use AttributeDescriptor rather than 
> introduce a new thing called "Prop" ?

You do know that Props are configuration properties, nothing to do with 
features?

> - Are the class responsibilities clear: 
> GranuleCatalogManager, GranuleCatalog, GranuleStore, MosaicHarvester, 
> etc...
>
> (It could also be that since I am unfamiliar with the internals being 
> refactored that I am just struggling with the language - so if 
> everyone else is okay ...)
>
>
Regards
Niels

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

>
> -1- Can you clarify this sentence?
>
> "to replace the default direct granule access with a renderable graph
> (for reprojection)"
>

Replaced with "support resampling of rasters into common CRS for mosaic
operation".


> -2- Stupid thing but in ImageMosaic I would go for add/removeRaster
> rather than *Coverage
>

Updated.

This proposal (and email thread) should be orthogonal to the RnD discussion
on setting up a mosaic with multiple projections.

This mosaic is focused on creating/managing the index used. One part I
cannot wrap my head around is the workflow behind these two methods
- createSchema(String name, CoordinateReferenceSystem crs) which creates
the FeatureType used for the index, and Collection
getPropertiesCollectors().

It seems these two methods need to be implemented together, a single
PropertyCollector may collect more than once value on examining a raster. I
am not sure how to line up these collected values with the created schema
without more information. We also have String getParameter(Prop prop)
defined, but the class Prop is not included in the proposal.

Suggestions:
- Use a list of PropertiesCollectors to provide order, which can then be
used to determine the Schema?
- Language consistency - raster / granule
- Language consistency - could we use AttributeDescriptor rather than
introduce a new thing called "Prop" ?
- Are the class responsibilities clear:
GranuleCatalogManager, GranuleCatalog, GranuleStore, MosaicHarvester, etc...

(It could also be that since I am unfamiliar with the internals being
refactored that I am just struggling with the language - so if everyone
else is okay ...)
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Niels Charlier]

Hi Simone,

Thanks for review! In response to your questions

(1) Jody added this note to the proposal and Devon is working on this 
part, so I think either one of them are best placed to explain this part.

(2) Okay, agreed.

Regards
Niels

On 15-04-16 10:39, Simone Giannecchini wrote:
> Ciao Niels,
> we are looking into this and comparing with some work that have in the
> pipeline which might extend the work you are doing with respect to
> harvesting.
>
> I like the general idea pretty much, I have a few quick questions/obs?
>
> -1- Can you clarify this sentence?
>
> "to replace the default direct granule access with a renderable graph
> (for reprojection)"
>
> -2- Stupid thing but in ImageMosaic I would go for add/removeRaster
> rather than *Coverage
>
> I will talk to Andrea and Daniele to add more.
>
> That said, it might be worth to have a look at the code when it start to 
> settle.
>
>
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Wed, Apr 6, 2016 at 10:33 AM, Niels Charlier  wrote:
>> Hello Devon,
>>
>> Directory walking is part of the harvesting process, which is at the moment
>> implemented inside the ImageMosaicReader (see the inner class
>> HarvestedResource).
>> In the API, there is a separate MosaicHarvester interface, as well as a very
>> light ImageMosaicReader with all the logic inside the ImageMosaic class.
>> With this API, we can either do one of these two thing to customise
>> harvesting:
>> 1. Write our own reader and use all the mosaic functionality through the
>> ImageMosaic class.
>> 2. Write our own implementation of MosaicHarvester for a customised
>> harvesting process and plug it into the ImageMosaicReader.
>>
>> Does that make any sense?
>>
>> Kind Regards
>> Niels
>>
>>
>> On 06-04-16 01:48, Devon Tucker wrote:
>>
>> For my own part, Niels maybe you could throw in some small examples of how
>> these API changes would be used in practice to accomplish some of the items
>> in the description? For example, how could I customize the directory walking
>> with the new API?
>>
>> Thanks,
>> Devon
>>
>> On Tue, Apr 5, 2016 at 4:44 PM, Devon Tucker  wrote:
>>> Hi all,
>>>
>>> Just noticed that there wasn't a dedicated email thread for this proposal
>>> and figured I'd kick one off since I've also been involved in it.
>>>
>>>
>>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>>
>>> I think the description gives a pretty good rationale for what we want to
>>> accomplish with this proposal and I'd like to solicit any discussion,
>>> advice, feedback, etc. from everyone.
>>>
>>> Cheers,
>>> Devon
>>
>>
>>
>> --
>>
>>
>>
>> ___
>> 

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Simone Giannecchini]

Ciao Niels,
we are looking into this and comparing with some work that have in the
pipeline which might extend the work you are doing with respect to
harvesting.

I like the general idea pretty much, I have a few quick questions/obs?

-1- Can you clarify this sentence?

"to replace the default direct granule access with a renderable graph
(for reprojection)"

-2- Stupid thing but in ImageMosaic I would go for add/removeRaster
rather than *Coverage

I will talk to Andrea and Daniele to add more.

That said, it might be worth to have a look at the code when it start to settle.



Regards,
Simone Giannecchini
==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==
Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob:   +39  333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility  for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.


On Wed, Apr 6, 2016 at 10:33 AM, Niels Charlier  wrote:
> Hello Devon,
>
> Directory walking is part of the harvesting process, which is at the moment
> implemented inside the ImageMosaicReader (see the inner class
> HarvestedResource).
> In the API, there is a separate MosaicHarvester interface, as well as a very
> light ImageMosaicReader with all the logic inside the ImageMosaic class.
> With this API, we can either do one of these two thing to customise
> harvesting:
> 1. Write our own reader and use all the mosaic functionality through the
> ImageMosaic class.
> 2. Write our own implementation of MosaicHarvester for a customised
> harvesting process and plug it into the ImageMosaicReader.
>
> Does that make any sense?
>
> Kind Regards
> Niels
>
>
> On 06-04-16 01:48, Devon Tucker wrote:
>
> For my own part, Niels maybe you could throw in some small examples of how
> these API changes would be used in practice to accomplish some of the items
> in the description? For example, how could I customize the directory walking
> with the new API?
>
> Thanks,
> Devon
>
> On Tue, Apr 5, 2016 at 4:44 PM, Devon Tucker  wrote:
>>
>> Hi all,
>>
>> Just noticed that there wasn't a dedicated email thread for this proposal
>> and figured I'd kick one off since I've also been involved in it.
>>
>>
>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>
>> I think the description gives a pretty good rationale for what we want to
>> accomplish with this proposal and I'd like to solicit any discussion,
>> advice, feedback, etc. from everyone.
>>
>> Cheers,
>> Devon
>
>
>
>
> --
>
>
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
>
> --
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Niels Charlier]

Hello Devon,

Directory walking is part of the harvesting process, which is at the 
moment implemented inside the ImageMosaicReader (see the inner class 
HarvestedResource).
In the API, there is a separate MosaicHarvester interface, as well as a 
very light ImageMosaicReader with all the logic inside the ImageMosaic 
class.
With this API, we can either do one of these two thing to customise 
harvesting:
1. Write our own reader and use all the mosaic functionality through the 
ImageMosaic class.
2. Write our own implementation of MosaicHarvester for a customised 
harvesting process and plug it into the ImageMosaicReader.


Does that make any sense?

Kind Regards
Niels

On 06-04-16 01:48, Devon Tucker wrote:
For my own part, Niels maybe you could throw in some small examples of 
how these API changes would be used in practice to accomplish some of 
the items in the description? For example, how could I customize the 
directory walking with the new API?


Thanks,
Devon

On Tue, Apr 5, 2016 at 4:44 PM, Devon Tucker > wrote:


Hi all,

Just noticed that there wasn't a dedicated email thread for this
proposal and figured I'd kick one off since I've also been
involved in it.


https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility

I think the description gives a pretty good rationale for what we
want to accomplish with this proposal and I'd like to solicit any
discussion, advice, feedback, etc. from everyone.

Cheers,
Devon




--


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

For my own part, Niels maybe you could throw in some small examples of how
these API changes would be used in practice to accomplish some of the items
in the description? For example, how could I customize the directory
walking with the new API?

Thanks,
Devon

On Tue, Apr 5, 2016 at 4:44 PM, Devon Tucker  wrote:

> Hi all,
>
> Just noticed that there wasn't a dedicated email thread for this proposal
> and figured I'd kick one off since I've also been involved in it.
>
>
> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>
> I think the description gives a pretty good rationale for what we want to
> accomplish with this proposal and I'd like to solicit any discussion,
> advice, feedback, etc. from everyone.
>
> Cheers,
> Devon
>
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] ImageMosaic API refactor proposal

[Devon Tucker]

Hi all,

Just noticed that there wasn't a dedicated email thread for this proposal
and figured I'd kick one off since I've also been involved in it.

https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility

I think the description gives a pretty good rationale for what we want to
accomplish with this proposal and I'd like to solicit any discussion,
advice, feedback, etc. from everyone.

Cheers,
Devon
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel