Re: [Geotools-devel] Using native filters in queries

[Nuno Oliveira]

Hi Jody,
thanks for the feedback, is quite interesting that we had the same thoughts ... my /native /function 
just

didn't had a /default/ option :P

Please see my answers bellow:

On 03/24/2018 10:40 AM, Jody Garnett wrote:
I think you have answers my question: we do have query hints for extra control over data store 
interaction, but for this purpose it is not connected close enough to filter encoding for your 
purpose.


I feel a bit odd opening a new kind of custom filter interface, when we have worked very hard not 
to do so in the past.


I understand that feeling ... We should keep in mind that this filter will provide an /extension 
point/ for

programmers not for the end user, which will \ should never have a direct 
contact with this filter.

This filter gives a chance to programmers to take full advantage of the 
underling data store when
needed in a transparent and harmonized way, this is not restricted to JDBC data 
stores.

... and at the end I don't see any concrete issue in adding this filter, unless 
some programmer
explicitly uses it this will have no impact on other things.



Other option you may of already considered is to handle this as a function that evaluates to 
true/false. Something like *native(query, fallback)*.  The first parameter would be SQL , the 
second a fallback value to be used when not run on a database.

- I guess this is just not optimizable enough since you are focused on filter 
splitters
- The true/false is not a very good contract, it would be handy to run a native 
expression
- The net effect would be similar to parametrized views in that any SLD using the filter or 
expression could still be portable (with reduced utility)




Yes I investigated this approach, and on top of what you say I can add this:

    1. The idea of  a native filter is that it should executed natively, so if it ends up being 
executed in

    memory the execution should fail, so I don't see the need for a default 
value.

    2. The other advantage of having a filter is that it is compatible with the existing stack, a 
function
    will need to have an explicit handling different form all the other functions. We should 
not focus

    only on the JDBC data store, we should keep in mind others data stores 
like App-Schema,
    Apache Solr, MongoDB, etc ...

   3. Another issue with a function is that it could end up being used by the end user, which would 
be a really
   bad thing. Just imagine all the possible security breaches. This Native filter instead, will 
not even be possible

   to encode it to XML or (E)CQL.

   4. Finally, even if we ignore all the issues above ... in an architectural point of view using a 
function for this
       would just be a major hack. Every time we do something with a filter (and there is quite a 
few places) we
   would have to check that the native function is present and handle it explicitly, because 
well it is not really

   a function.

Cheers,

Nuno Oliveira



On Wed, Mar 21, 2018 at 11:08 AM Nuno Oliveira > wrote:


Hi Jody,

No sure I understand your comment:

 1. you mean there is already something similar that can be done with query 
hints ?
 2. or you mean we could use a query hint for this ?

That say, one my first approaches was to take advantage of query hints but 
I step on to many
limitations, just to /name/ a few:

  * query hints at best allows us to say, please happen this to the final 
query that will be
send to the data store, but doesn't allow us to build a filter the way 
we need it (OR,
AND, nested, etc ...)

  * the native filter provides a more harmonized approach, the query hint 
would have to be
managed explicitly in every part of the code where the final query that 
will be send to
the data store is created, this would quickly become a nightmare (note 
this is not
specific to JDBC data store)

  * having a filter allows us to take advantage of all the existing wiring 
and processing that
involves filters visitors, e.g. capabilities support, filters splitter, 
etc ...

Another thing that is worth mentioning, is that for most of the use cases I 
can think of being
able to use a filter is more /compatible/ for the invokers than setting up 
a query hint.

Kind regards,

Nuno Oliveira


On 03/21/2018 01:59 AM, Jody Garnett wrote:

do we not have query hints for this purpose?
On Tue, Mar 20, 2018 at 12:07 PM Nuno Oliveira 
> wrote:

Hi Andrea,
thanks for the feedback, please see my answers bellow:


On 03/20/2018 10:21 AM, Andrea Aime wrote:

Hi Nuno,
last time a new type of filter was in this commit:


https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf
   

Re: [Geotools-devel] Using native filters in queries

[Jody Garnett]

I think you have answers my question: we do have query hints for extra
control over data store interaction, but for this purpose it is not
connected close enough to filter encoding for your purpose.

I feel a bit odd opening a new kind of custom filter interface, when we
have worked very hard not to do so in the past.

Other option you may of already considered is to handle this as a function
that evaluates to true/false. Something like *native(query, fallback)*.  The
first parameter would be SQL , the second a fallback value to be used when
not run on a database.
- I guess this is just not optimizable enough since you are focused on
filter splitters
- The true/false is not a very good contract, it would be handy to run a
native expression
- The net effect would be similar to parametrized views in that any SLD
using the filter or expression could still be portable (with reduced
utility)


On Wed, Mar 21, 2018 at 11:08 AM Nuno Oliveira  wrote:

> Hi Jody,
>
> No sure I understand your comment:
>
>1. you mean there is already something similar that can be done with
>query hints ?
>2. or you mean we could use a query hint for this ?
>
> That say, one my first approaches was to take advantage of query hints but
> I step on to many limitations, just to *name* a few:
>
>- query hints at best allows us to say, please happen this to the
>final query that will be send to the data store, but doesn't allow us to
>build a filter the way we need it (OR, AND, nested, etc ...)
>
>
>- the native filter provides a more harmonized approach, the query
>hint would have to be managed explicitly in every part of the code where
>the final query that will be send to the data store is created, this would
>quickly become a nightmare (note this is not specific to JDBC data store)
>
>
>- having a filter allows us to take advantage of all the existing
>wiring and processing that involves filters visitors, e.g. capabilities
>support, filters splitter, etc ...
>
> Another thing that is worth mentioning, is that for most of the use cases
> I can think of being able to use a filter is more *compatible* for the
> invokers than setting up a query hint.
> Kind regards,
>
> Nuno Oliveira
>
>
> On 03/21/2018 01:59 AM, Jody Garnett wrote:
>
> do we not have query hints for this purpose?
> On Tue, Mar 20, 2018 at 12:07 PM Nuno Oliveira <
> nuno.olive...@geo-solutions.it> wrote:
>
>> Hi Andrea,
>> thanks for the feedback, please see my answers bellow:
>>
>>
>> On 03/20/2018 10:21 AM, Andrea Aime wrote:
>>
>> Hi Nuno,
>> last time a new type of filter was in this commit:
>>
>> https://github.com/geotools/geotools/commit/
>> 47e9259d74482b92244b4dd301363b434e382baf
>>
>> Thankfully now there are default methods, allowing for quite a bit less
>> painful extension.
>>
>>
>> Thanks, yes default methods help indeed.
>>
>>
>> Overall I see where this need comes from and agree with it, a few
>> notes/questions off the top of my head:
>>
>>- Can you share how you intend to implement NativeFilter? Is it an
>>interface with store specific implementations, or does it have a single
>>one? Which stores will receive a sample implementation of the concept?
>>
>>
>> My idea is to create an interface (this will help with filter visitors):
>>
>> public interface NativeFilter extends Filter {
>>
>> Expression getNative();
>> }
>>
>>  and then provide a default implementation for it, this default
>> implementation should be enough for most use cases.
>>
>> The JDBC data store will be the first one receiving support for it.
>>
>>
>>
>>- Make sure the functionality is open for programmers but not from
>>the web (e.g., one cannot setup a XML or CQL filter using it
>>
>>
>> Yes this is an important point indeed, no XML binds or CQL support will
>> be implemented. The Filter to CQL visitor will throw an exception.
>>
>>
>>
>>- Properly document the intended usage of it including notion of
>>pre/post filter splitting, e.g., if a NativeFilter is supported, but it's
>>OR-ed with another bit using an unsupported filter function, it will end 
>> in
>>the post, in memory evaluation, and will blow up.This is unavoidable, but
>>should not be a surprise for users.
>>
>>
>> Yep.
>>
>>
>> Cheers
>> Andrea
>>
>>
>>
>>
>>
>> On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira <
>> nuno.olive...@geo-solutions.it> wrote:
>>
>>> Hi all,
>>>
>>> Recently I have been dealing with some use cases that require the
>>> ability to inject some native filters,
>>> specific for each feature type and non static over time, to the final
>>> native query build by GeoTools and
>>> send to the data store.
>>>
>>> For example, the content of a feature type is partially controlled by an
>>> external service who provides a non
>>> static native query (the request changes over time) that filters the
>>> content that should be show.
>>>
>>> Another use case is a complex authorization 

Re: [Geotools-devel] Using native filters in queries

[Andrea Aime]

Hi,
adding my 2 cents, one more distinction that is kind of important given
Nuno's use case, that is security related:
an unknown query hint will just be ignored (bad, no security applied), a
unknown filter is normally going to throw an exception somewhere (worst
case, when it's being evaluated in memory), so normally no unsecured data
would be returned.

Cheers
Andrea


On Wed, Mar 21, 2018 at 11:08 AM, Nuno Oliveira <
nuno.olive...@geo-solutions.it> wrote:

> Hi Jody,
>
> No sure I understand your comment:
>
>1. you mean there is already something similar that can be done with
>query hints ?
>2. or you mean we could use a query hint for this ?
>
> That say, one my first approaches was to take advantage of query hints but
> I step on to many limitations, just to *name* a few:
>
>- query hints at best allows us to say, please happen this to the
>final query that will be send to the data store, but doesn't allow us to
>build a filter the way we need it (OR, AND, nested, etc ...)
>
>
>- the native filter provides a more harmonized approach, the query
>hint would have to be managed explicitly in every part of the code where
>the final query that will be send to the data store is created, this would
>quickly become a nightmare (note this is not specific to JDBC data store)
>
>
>- having a filter allows us to take advantage of all the existing
>wiring and processing that involves filters visitors, e.g. capabilities
>support, filters splitter, etc ...
>
> Another thing that is worth mentioning, is that for most of the use cases
> I can think of being able to use a filter is more *compatible* for the
> invokers than setting up a query hint.
> Kind regards,
>
> Nuno Oliveira
>
>
> On 03/21/2018 01:59 AM, Jody Garnett wrote:
>
> do we not have query hints for this purpose?
> On Tue, Mar 20, 2018 at 12:07 PM Nuno Oliveira <
> nuno.olive...@geo-solutions.it> wrote:
>
>> Hi Andrea,
>> thanks for the feedback, please see my answers bellow:
>>
>>
>> On 03/20/2018 10:21 AM, Andrea Aime wrote:
>>
>> Hi Nuno,
>> last time a new type of filter was in this commit:
>>
>> https://github.com/geotools/geotools/commit/
>> 47e9259d74482b92244b4dd301363b434e382baf
>>
>> Thankfully now there are default methods, allowing for quite a bit less
>> painful extension.
>>
>>
>> Thanks, yes default methods help indeed.
>>
>>
>> Overall I see where this need comes from and agree with it, a few
>> notes/questions off the top of my head:
>>
>>- Can you share how you intend to implement NativeFilter? Is it an
>>interface with store specific implementations, or does it have a single
>>one? Which stores will receive a sample implementation of the concept?
>>
>>
>> My idea is to create an interface (this will help with filter visitors):
>>
>> public interface NativeFilter extends Filter {
>>
>> Expression getNative();
>> }
>>
>>  and then provide a default implementation for it, this default
>> implementation should be enough for most use cases.
>>
>> The JDBC data store will be the first one receiving support for it.
>>
>>
>>
>>- Make sure the functionality is open for programmers but not from
>>the web (e.g., one cannot setup a XML or CQL filter using it
>>
>>
>> Yes this is an important point indeed, no XML binds or CQL support will
>> be implemented. The Filter to CQL visitor will throw an exception.
>>
>>
>>
>>- Properly document the intended usage of it including notion of
>>pre/post filter splitting, e.g., if a NativeFilter is supported, but it's
>>OR-ed with another bit using an unsupported filter function, it will end 
>> in
>>the post, in memory evaluation, and will blow up.This is unavoidable, but
>>should not be a surprise for users.
>>
>>
>> Yep.
>>
>>
>> Cheers
>> Andrea
>>
>>
>>
>>
>>
>> On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira <
>> nuno.olive...@geo-solutions.it> wrote:
>>
>>> Hi all,
>>>
>>> Recently I have been dealing with some use cases that require the
>>> ability to inject some native filters,
>>> specific for each feature type and non static over time, to the final
>>> native query build by GeoTools and
>>> send to the data store.
>>>
>>> For example, the content of a feature type is partially controlled by an
>>> external service who provides a non
>>> static native query (the request changes over time) that filters the
>>> content that should be show.
>>>
>>> Another use case is a complex authorization rules system where the
>>> authorization rules are SQL
>>> expressions that need to be append to the final SQL WHERE clause build
>>> by GeoTools, the system
>>> provides SQL expressions for different databases (Oracle, PostgreSQL,
>>> MySQL, etc ...).
>>>
>>> Most of the native filters of this use cases are quite complex and can't
>>> be translated to OGC filters.
>>>
>>> Since we have a pattern here that can be applied to several use cases
>>> and data stores I have think about
>>> an approach that would allow 

Re: [Geotools-devel] Using native filters in queries

[Nuno Oliveira]

Hi Jody,

No sure I understand your comment:

1. you mean there is already something similar that can be done with query 
hints ?
2. or you mean we could use a query hint for this ?

That say, one my first approaches was to take advantage of query hints but I step on to many 
limitations, just to /name/// a few:


 * query hints at best allows us to say, please happen this to the final query 
that will be send to
   the data store, but doesn't allow us to build a filter the way we need it 
(OR, AND, nested, etc ...)

 * the native filter provides a more harmonized approach, the query hint would 
have to be managed
   explicitly in every part of the code where the final query that will be send 
to the data store
   is created, this would quickly become a nightmare (note this is not specific 
to JDBC data store)

 * having a filter allows us to take advantage of all the existing wiring and 
processing that
   involves filters visitors, e.g. capabilities support, filters splitter, etc 
...

Another thing that is worth mentioning, is that for most of the use cases I can think of being able 
to use a filter is more /compatible/ for the invokers than setting up a query hint.


Kind regards,

Nuno Oliveira

On 03/21/2018 01:59 AM, Jody Garnett wrote:

do we not have query hints for this purpose?
On Tue, Mar 20, 2018 at 12:07 PM Nuno Oliveira > wrote:


Hi Andrea,
thanks for the feedback, please see my answers bellow:


On 03/20/2018 10:21 AM, Andrea Aime wrote:

Hi Nuno,
last time a new type of filter was in this commit:


https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf

Thankfully now there are default methods, allowing for quite a bit less 
painful extension.


Thanks, yes default methods help indeed.



Overall I see where this need comes from and agree with it, a few 
notes/questions off the top
of my head:

  * Can you share how you intend to implement NativeFilter? Is it an 
interface with store
specific implementations, or does it have a single one? Which stores 
will receive a
sample implementation of the concept?



My idea is to create an interface (this will help with filter visitors):

public interface NativeFilter extends Filter {

    Expression getNative();
}

 and then provide a default implementation for it, this default 
implementation should be
enough for most use cases.

The JDBC data store will be the first one receiving support for it.



  * Make sure the functionality is open for programmers but not from the 
web (e.g., one
cannot setup a XML or CQL filter using it



Yes this is an important point indeed, no XML binds or CQL support will be 
implemented. The
Filter to CQL visitor will throw an exception.



  * Properly document the intended usage of it including notion of pre/post 
filter splitting,
e.g., if a NativeFilter is supported, but it's OR-ed with another bit 
using an
unsupported filter function, it will end in the post, in memory 
evaluation, and will blow
up.This is unavoidable, but should not be a surprise for users.



Yep.



Cheers
Andrea





On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira 
> wrote:

Hi all,

Recently I have been dealing with some use cases that require the 
ability to inject some
native filters,
specific for each feature type and non static over time, to the final 
native query build
by GeoTools and
send to the data store.

For example, the content of a feature type is partially controlled by 
an external service
who provides a non
static native query (the request changes over time) that filters the 
content that should
be show.

Another use case is a complex authorization rules system where the 
authorization rules
are SQL
expressions that need to be append to the final SQL WHERE clause build 
by GeoTools, the
system
provides SQL expressions for different databases (Oracle, PostgreSQL, 
MySQL, etc ...).

Most of the native filters of this use cases are quite complex and 
can't be translated to
OGC filters.

Since we have a pattern here that can be applied to several use cases 
and data stores I
have think about
an approach that would allow extensions, external code, etc ... to 
inject the necessary
native filter(s)
and allow data stores implementations to handle those native filters in 
an harmonized way.

I have come up with the idea of a new filter named /NativeFilter/:

  * this filter will NOT be directly usable by the final users, for 
example in WFS requests

  * if any code try to evaluate it in memory an 

Re: [Geotools-devel] Using native filters in queries

[Jody Garnett]

do we not have query hints for this purpose?
On Tue, Mar 20, 2018 at 12:07 PM Nuno Oliveira <
nuno.olive...@geo-solutions.it> wrote:

> Hi Andrea,
> thanks for the feedback, please see my answers bellow:
>
>
> On 03/20/2018 10:21 AM, Andrea Aime wrote:
>
> Hi Nuno,
> last time a new type of filter was in this commit:
>
>
> https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf
>
> Thankfully now there are default methods, allowing for quite a bit less
> painful extension.
>
>
> Thanks, yes default methods help indeed.
>
>
> Overall I see where this need comes from and agree with it, a few
> notes/questions off the top of my head:
>
>- Can you share how you intend to implement NativeFilter? Is it an
>interface with store specific implementations, or does it have a single
>one? Which stores will receive a sample implementation of the concept?
>
>
> My idea is to create an interface (this will help with filter visitors):
>
> public interface NativeFilter extends Filter {
>
> Expression getNative();
> }
>
>  and then provide a default implementation for it, this default
> implementation should be enough for most use cases.
>
> The JDBC data store will be the first one receiving support for it.
>
>
>
>- Make sure the functionality is open for programmers but not from the
>web (e.g., one cannot setup a XML or CQL filter using it
>
>
> Yes this is an important point indeed, no XML binds or CQL support will be
> implemented. The Filter to CQL visitor will throw an exception.
>
>
>
>- Properly document the intended usage of it including notion of
>pre/post filter splitting, e.g., if a NativeFilter is supported, but it's
>OR-ed with another bit using an unsupported filter function, it will end in
>the post, in memory evaluation, and will blow up.This is unavoidable, but
>should not be a surprise for users.
>
>
> Yep.
>
>
> Cheers
> Andrea
>
>
>
>
>
> On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira <
> nuno.olive...@geo-solutions.it> wrote:
>
>> Hi all,
>>
>> Recently I have been dealing with some use cases that require the ability
>> to inject some native filters,
>> specific for each feature type and non static over time, to the final
>> native query build by GeoTools and
>> send to the data store.
>>
>> For example, the content of a feature type is partially controlled by an
>> external service who provides a non
>> static native query (the request changes over time) that filters the
>> content that should be show.
>>
>> Another use case is a complex authorization rules system where the
>> authorization rules are SQL
>> expressions that need to be append to the final SQL WHERE clause build by
>> GeoTools, the system
>> provides SQL expressions for different databases (Oracle, PostgreSQL,
>> MySQL, etc ...).
>>
>> Most of the native filters of this use cases are quite complex and can't
>> be translated to OGC filters.
>>
>> Since we have a pattern here that can be applied to several use cases and
>> data stores I have think about
>> an approach that would allow extensions, external code, etc ... to inject
>> the necessary native filter(s)
>> and allow data stores implementations to handle those native filters in
>> an harmonized way.
>>
>> I have come up with the idea of a new filter named *NativeFilter*:
>>
>>- this filter will NOT be directly usable by the final users, for
>>example in WFS requests
>>
>>
>>- if any code try to evaluate it in memory an exception will be throw
>>
>>
>>- it will not break any API and data stores will be free to implement
>>is support has needed
>>
>>
>>- it will have a method getNative() that will return a String
>>containing the native filter
>>
>> A new default method would need to be added to the *FilterVisitor*
>> interface and a default
>> implementation would need to be added to *AsbtractFilterVisitor and
>> DuplicatingFilterVisitor *
>> classes (like for any other filter).
>>
>> The data store who wants to supports this filter will need to extends it
>> custom visitor, for example,
>> for relational databases we only need to add a new visit method to
>> *FilterToSQL* visitor.
>>
>> This gives the necessary flexibility for third party code (codes that use
>> GeoTools) to inject native
>> filters without any specific extension points or cumbersome handling. It
>> is also worth notice unless
>> some code is using native filters this will have no impact on other
>> usages.
>>
>> Any thoughts on this ? Objections ?
>>
>> King regards,
>>
>> Nuno Oliveira
>>
>> --
>> Regards,
>> Nuno Oliveira
>> ==
>> GeoServer Professional Services from the experts! Visit http://goo.gl/it488V 
>> for more information.
>> ==
>>
>> Nuno Miguel Carvalho Oliveira
>> @nmcoliveira
>> Software Engineer
>>
>> GeoSolutions S.A.S.
>> Via di Montramito 3/A
>> 55054  Massarosa (LU)
>> Italy
>> phone: +39 0584 962313 <+39%200584%20962313>
>> fax:  +39 0584 1660272 <+39%200584%20166%200272>
>> 

Re: [Geotools-devel] Using native filters in queries

[Nuno Oliveira]

Hi Andrea,
thanks for the feedback, please see my answers bellow:

On 03/20/2018 10:21 AM, Andrea Aime wrote:

Hi Nuno,
last time a new type of filter was in this commit:

https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf

Thankfully now there are default methods, allowing for quite a bit less painful 
extension.


Thanks, yes default methods help indeed.

Overall I see where this need comes from and agree with it, a few notes/questions off the top of 
my head:


  * Can you share how you intend to implement NativeFilter? Is it an interface 
with store specific
implementations, or does it have a single one? Which stores will receive a 
sample
implementation of the concept?



My idea is to create an interface (this will help with filter visitors):

   public interface NativeFilter extends Filter {

    Expression getNative();
   }

 and then provide a default implementation for it, this default implementation should be enough for 
most use cases.


The JDBC data store will be the first one receiving support for it.


  * Make sure the functionality is open for programmers but not from the web 
(e.g., one cannot
setup a XML or CQL filter using it



Yes this is an important point indeed, no XML binds or CQL support will be implemented. The Filter 
to CQL visitor will throw an exception.



  * Properly document the intended usage of it including notion of pre/post 
filter splitting,
e.g., if a NativeFilter is supported, but it's OR-ed with another bit using 
an unsupported
filter function, it will end in the post, in memory evaluation, and will 
blow up.This is
unavoidable, but should not be a surprise for users.



Yep.


Cheers
Andrea





On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira > wrote:


Hi all,

Recently I have been dealing with some use cases that require the ability 
to inject some
native filters,
specific for each feature type and non static over time, to the final 
native query build by
GeoTools and
send to the data store.

For example, the content of a feature type is partially controlled by an 
external service who
provides a non
static native query (the request changes over time) that filters the 
content that should be show.

Another use case is a complex authorization rules system where the 
authorization rules are SQL
expressions that need to be append to the final SQL WHERE clause build by 
GeoTools, the system
provides SQL expressions for different databases (Oracle, PostgreSQL, 
MySQL, etc ...).

Most of the native filters of this use cases are quite complex and can't be 
translated to OGC
filters.

Since we have a pattern here that can be applied to several use cases and 
data stores I have
think about
an approach that would allow extensions, external code, etc ... to inject 
the necessary native
filter(s)
and allow data stores implementations to handle those native filters in an 
harmonized way.

I have come up with the idea of a new filter named /NativeFilter/:

  * this filter will NOT be directly usable by the final users, for example 
in WFS requests

  * if any code try to evaluate it in memory an exception will be throw

  * it will not break any API and data stores will be free to implement is 
support has needed

  * it will have a method getNative() that will return a String containing 
the native filter

A new default method would need to be added to the /FilterVisitor/ 
interface and a default
implementation would need to be added to /AsbtractFilterVisitor and 
DuplicatingFilterVisitor /
classes (like for any other filter).

The data store who wants to supports this filter will need to extends it 
custom visitor, for
example,
for relational databases we only need to add a new visit method to 
/FilterToSQL/ visitor.

This gives the necessary flexibility for third party code (codes that use 
GeoTools) to inject
native
filters without any specific extension points or cumbersome handling. It is 
also worth notice
unless
some code is using native filters this will have no impact on other usages.

Any thoughts on this ? Objections ?

King regards,

Nuno Oliveira

-- 
Regards,

Nuno Oliveira
==
GeoServer Professional Services from the experts! Visithttp://goo.gl/it488V 
 for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone:+39 0584 962313 
fax:+39 0584 1660272 

http://www.geo-solutions.it
http://twitter.com/geosolutions_it 

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
 

Re: [Geotools-devel] Using native filters in queries

[Andrea Aime]

Hi Nuno,
last time a new type of filter was in this commit:

https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf

Thankfully now there are default methods, allowing for quite a bit less
painful extension.
Overall I see where this need comes from and agree with it, a few
notes/questions off the top of my head:

   - Can you share how you intend to implement NativeFilter? Is it an
   interface with store specific implementations, or does it have a single
   one? Which stores will receive a sample implementation of the concept?
   - Make sure the functionality is open for programmers but not from the
   web (e.g., one cannot setup a XML or CQL filter using it
   - Properly document the intended usage of it including notion of
   pre/post filter splitting, e.g., if a NativeFilter is supported, but it's
   OR-ed with another bit using an unsupported filter function, it will end in
   the post, in memory evaluation, and will blow up.This is unavoidable, but
   should not be a surprise for users.

Cheers
Andrea





On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira <
nuno.olive...@geo-solutions.it> wrote:

> Hi all,
>
> Recently I have been dealing with some use cases that require the ability
> to inject some native filters,
> specific for each feature type and non static over time, to the final
> native query build by GeoTools and
> send to the data store.
>
> For example, the content of a feature type is partially controlled by an
> external service who provides a non
> static native query (the request changes over time) that filters the
> content that should be show.
>
> Another use case is a complex authorization rules system where the
> authorization rules are SQL
> expressions that need to be append to the final SQL WHERE clause build by
> GeoTools, the system
> provides SQL expressions for different databases (Oracle, PostgreSQL,
> MySQL, etc ...).
>
> Most of the native filters of this use cases are quite complex and can't
> be translated to OGC filters.
>
> Since we have a pattern here that can be applied to several use cases and
> data stores I have think about
> an approach that would allow extensions, external code, etc ... to inject
> the necessary native filter(s)
> and allow data stores implementations to handle those native filters in an
> harmonized way.
>
> I have come up with the idea of a new filter named *NativeFilter*:
>
>- this filter will NOT be directly usable by the final users, for
>example in WFS requests
>
>
>- if any code try to evaluate it in memory an exception will be throw
>
>
>- it will not break any API and data stores will be free to implement
>is support has needed
>
>
>- it will have a method getNative() that will return a String
>containing the native filter
>
> A new default method would need to be added to the *FilterVisitor*
> interface and a default
> implementation would need to be added to *AsbtractFilterVisitor and
> DuplicatingFilterVisitor *
> classes (like for any other filter).
>
> The data store who wants to supports this filter will need to extends it
> custom visitor, for example,
> for relational databases we only need to add a new visit method to
> *FilterToSQL* visitor.
>
> This gives the necessary flexibility for third party code (codes that use
> GeoTools) to inject native
> filters without any specific extension points or cumbersome handling. It
> is also worth notice unless
> some code is using native filters this will have no impact on other usages.
>
> Any thoughts on this ? Objections ?
>
> King regards,
>
> Nuno Oliveira
>
> --
> Regards,
> Nuno Oliveira
> ==
> GeoServer Professional Services from the experts! Visit http://goo.gl/it488V 
> for more information.
> ==
>
> Nuno Miguel Carvalho Oliveira
> @nmcoliveira
> Software Engineer
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313 <+39%200584%20962313>
> fax:  +39 0584 1660272 <+39%200584%20166%200272>
> http://www.geo-solutions.ithttp://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i 
> file/s allegato/i sono da considerarsi strettamente riservate. Il loro 
> utilizzo è consentito esclusivamente al destinatario del messaggio, per le 
> finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio 
> senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia 
> via e-mail e di procedere alla distruzione del messaggio stesso, 
> cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo 
> anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per 
> finalità diverse, costituisce comportamento contrario ai principi dettati dal 
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely for 
> the attention and use of the 

[Geotools-devel] Using native filters in queries

[Nuno Oliveira]

Hi all,

Recently I have been dealing with some use cases that require the ability to inject some native 
filters,
specific for each feature type and non static over time, to the final native query build by GeoTools 
and

send to the data store.

For example, the content of a feature type is partially controlled by an external service who 
provides a non

static native query (the request changes over time) that filters the content 
that should be show.

Another use case is a complex authorization rules system where the 
authorization rules are SQL
expressions that need to be append to the final SQL WHERE clause build by 
GeoTools, the system
provides SQL expressions for different databases (Oracle, PostgreSQL, MySQL, 
etc ...).

Most of the native filters of this use cases are quite complex and can't be 
translated to OGC filters.

Since we have a pattern here that can be applied to several use cases and data stores I have think 
about
an approach that would allow extensions, external code, etc ... to inject the necessary native 
filter(s)

and allow data stores implementations to handle those native filters in an 
harmonized way.

I have come up with the idea of a new filter named /NativeFilter/:

 * this filter will NOT be directly usable by the final users, for example in 
WFS requests

 * if any code try to evaluate it in memory an exception will be throw

 * it will not break any API and data stores will be free to implement is 
support has needed

 * it will have a method getNative() that will return a String containing the 
native filter

A new default method would need to be added to the /FilterVisitor/ interface 
and a default
implementation would need to be added to /AsbtractFilterVisitor and 
DuplicatingFilterVisitor /
classes (like for any other filter).

The data store who wants to supports this filter will need to extends it custom 
visitor, for example,
for relational databases we only need to add a new visit method to 
/FilterToSQL/ visitor.

This gives the necessary flexibility for third party code (codes that use 
GeoTools) to inject native
filters without any specific extension points or cumbersome handling. It is 
also worth notice unless
some code is using native filters this will have no impact on other usages.

Any thoughts on this ? Objections ?

King regards,

Nuno Oliveira

--
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V 
for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:  +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i 
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo 
è consentito esclusivamente al destinatario del messaggio, per le finalità 
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne 
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di 
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro 
sistema. Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, 
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
 
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel