Hi Jody,
thanks for the feedback, is quite interesting that we had the same thoughts ... my /native /function
just
didn't had a /default/ option :P
Please see my answers bellow:
On 03/24/2018 10:40 AM, Jody Garnett wrote:
I think you have answers my question: we do have query hints for extra control over data store
interaction, but for this purpose it is not connected close enough to filter encoding for your
purpose.
I feel a bit odd opening a new kind of custom filter interface, when we have worked very hard not
to do so in the past.
I understand that feeling ... We should keep in mind that this filter will provide an /extension
point/ for
programmers not for the end user, which will \ should never have a direct
contact with this filter.
This filter gives a chance to programmers to take full advantage of the
underling data store when
needed in a transparent and harmonized way, this is not restricted to JDBC data
stores.
... and at the end I don't see any concrete issue in adding this filter, unless
some programmer
explicitly uses it this will have no impact on other things.
Other option you may of already considered is to handle this as a function that evaluates to
true/false. Something like *native(query, fallback)*. The first parameter would be SQL , the
second a fallback value to be used when not run on a database.
- I guess this is just not optimizable enough since you are focused on filter
splitters
- The true/false is not a very good contract, it would be handy to run a native
expression
- The net effect would be similar to parametrized views in that any SLD using the filter or
expression could still be portable (with reduced utility)
Yes I investigated this approach, and on top of what you say I can add this:
1. The idea of a native filter is that it should executed natively, so if it ends up being
executed in
memory the execution should fail, so I don't see the need for a default
value.
2. The other advantage of having a filter is that it is compatible with the existing stack, a
function
will need to have an explicit handling different form all the other functions. We should
not focus
only on the JDBC data store, we should keep in mind others data stores
like App-Schema,
Apache Solr, MongoDB, etc ...
3. Another issue with a function is that it could end up being used by the end user, which would
be a really
bad thing. Just imagine all the possible security breaches. This Native filter instead, will
not even be possible
to encode it to XML or (E)CQL.
4. Finally, even if we ignore all the issues above ... in an architectural point of view using a
function for this
would just be a major hack. Every time we do something with a filter (and there is quite a
few places) we
would have to check that the native function is present and handle it explicitly, because
well it is not really
a function.
Cheers,
Nuno Oliveira
On Wed, Mar 21, 2018 at 11:08 AM Nuno Oliveira <nuno.olive...@geo-solutions.it
<mailto:nuno.olive...@geo-solutions.it>> wrote:
Hi Jody,
No sure I understand your comment:
1. you mean there is already something similar that can be done with query
hints ?
2. or you mean we could use a query hint for this ?
That say, one my first approaches was to take advantage of query hints but
I step on to many
limitations, just to /name/ a few:
* query hints at best allows us to say, please happen this to the final
query that will be
send to the data store, but doesn't allow us to build a filter the way
we need it (OR,
AND, nested, etc ...)
* the native filter provides a more harmonized approach, the query hint
would have to be
managed explicitly in every part of the code where the final query that
will be send to
the data store is created, this would quickly become a nightmare (note
this is not
specific to JDBC data store)
* having a filter allows us to take advantage of all the existing wiring
and processing that
involves filters visitors, e.g. capabilities support, filters splitter,
etc ...
Another thing that is worth mentioning, is that for most of the use cases I
can think of being
able to use a filter is more /compatible/ for the invokers than setting up
a query hint.
Kind regards,
Nuno Oliveira
On 03/21/2018 01:59 AM, Jody Garnett wrote:
do we not have query hints for this purpose?
On Tue, Mar 20, 2018 at 12:07 PM Nuno Oliveira
<nuno.olive...@geo-solutions.it
<mailto:nuno.olive...@geo-solutions.it>> wrote:
Hi Andrea,
thanks for the feedback, please see my answers bellow:
On 03/20/2018 10:21 AM, Andrea Aime wrote:
Hi Nuno,
last time a new type of filter was in this commit:
https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf
<https://github.com/geotools/geotools/commit/47e9259d74482b92244b4dd301363b434e382baf>
Thankfully now there are default methods, allowing for quite a bit less
painful extension.
Thanks, yes default methods help indeed.
Overall I see where this need comes from and agree with it, a few
notes/questions off
the top of my head:
* Can you share how you intend to implement NativeFilter? Is it an
interface with
store specific implementations, or does it have a single one? Which
stores will
receive a sample implementation of the concept?
My idea is to create an interface (this will help with filter visitors):
public interface NativeFilter extends Filter {
Expression getNative();
}
and then provide a default implementation for it, this default
implementation should be
enough for most use cases.
The JDBC data store will be the first one receiving support for it.
* Make sure the functionality is open for programmers but not from
the web (e.g., one
cannot setup a XML or CQL filter using it
Yes this is an important point indeed, no XML binds or CQL support will
be implemented.
The Filter to CQL visitor will throw an exception.
* Properly document the intended usage of it including notion of
pre/post filter
splitting, e.g., if a NativeFilter is supported, but it's OR-ed
with another bit
using an unsupported filter function, it will end in the post, in
memory evaluation,
and will blow up.This is unavoidable, but should not be a surprise
for users.
Yep.
Cheers
Andrea
On Mon, Mar 19, 2018 at 11:07 PM, Nuno Oliveira
<nuno.olive...@geo-solutions.it
<mailto:nuno.olive...@geo-solutions.it>> wrote:
Hi all,
Recently I have been dealing with some use cases that require the
ability to inject
some native filters,
specific for each feature type and non static over time, to the
final native query
build by GeoTools and
send to the data store.
For example, the content of a feature type is partially controlled
by an external
service who provides a non
static native query (the request changes over time) that filters
the content that
should be show.
Another use case is a complex authorization rules system where the
authorization
rules are SQL
expressions that need to be append to the final SQL WHERE clause
build by GeoTools,
the system
provides SQL expressions for different databases (Oracle,
PostgreSQL, MySQL, etc ...).
Most of the native filters of this use cases are quite complex and
can't be
translated to OGC filters.
Since we have a pattern here that can be applied to several use
cases and data
stores I have think about
an approach that would allow extensions, external code, etc ... to
inject the
necessary native filter(s)
and allow data stores implementations to handle those native
filters in an
harmonized way.
I have come up with the idea of a new filter named /NativeFilter/:
* this filter will NOT be directly usable by the final users, for
example in WFS
requests
* if any code try to evaluate it in memory an exception will be
throw
* it will not break any API and data stores will be free to
implement is support
has needed
* it will have a method getNative() that will return a String
containing the
native filter
A new default method would need to be added to the /FilterVisitor/
interface and a
default
implementation would need to be added to /AsbtractFilterVisitor and
DuplicatingFilterVisitor /
classes (like for any other filter).
The data store who wants to supports this filter will need to
extends it custom
visitor, for example,
for relational databases we only need to add a new visit method to
/FilterToSQL/
visitor.
This gives the necessary flexibility for third party code (codes
that use GeoTools)
to inject native
filters without any specific extension points or cumbersome
handling. It is also
worth notice unless
some code is using native filters this will have no impact on other
usages.
Any thoughts on this ? Objections ?
King regards,
Nuno Oliveira
--
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the experts!
Visithttp://goo.gl/it488V for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer
GeoSolutions S.A.S.
Via di Montramito 3/A 55054 Massarosa
<https://maps.google.com/?q=Via+di+Montramito+3/A%0A55054++Massarosa&entry=gmail&source=g>
(LU)
Italy
phone:+39 0584 962313 <tel:+39%200584%20962313>
fax:+39 0584 1660272 <tel:+39%200584%20166%200272>
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
<http://twitter.com/geosolutions_it>
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica
e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le
finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio
senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal
Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel
<https://lists.sourceforge.net/lists/listinfo/geotools-devel>
--
Regards,
Andrea Aime
==GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more
information.==Ing. Andrea Aime
<https://maps.google.com/?q=Via+di+Montramito+3/A+%0A+55054+%C2%A0Massarosa&entry=gmail&source=g>@geowolfTechnical
LeadGeoSolutions S.A.S.Via di Montramito 3/A
<https://maps.google.com/?q=Via+di+Montramito+3/A+%0A+55054+%C2%A0Massarosa&entry=gmail&source=g>55054
Massarosa
<https://maps.google.com/?q=Via+di+Montramito+3/A+%0A+55054+%C2%A0Massarosa&entry=gmail&source=g>
(LU)phone: +39 0584 962313 <tel:+39%200584%20962313>fax: +39 0584
1660272
<tel:+39%200584%20166%200272>mob: +39 339 8844549
<tel:+39%20339%20884%204549>http://www.geo-solutions.ithttp://twitter.com/geosolutions_it
<http://twitter.com/geosolutions_it>
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s
allegato/i sono da considerarsi strettamente riservate. Il loro
utilizzo è consentito
esclusivamente al destinatario del messaggio, per le finalità indicate
nel messaggio
stesso. Qualora riceviate questo messaggio senza esserne il
destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio
stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely
for the attention
and use of the named addressee(s) and may be confidential or
proprietary in nature or
covered by the provisions of privacy act (Legislative Decree June, 30
2003, no.196 -
Italy's New Data Protection Code).Any use not in accord with its
purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole
or partial, is strictly forbidden except previous formal approval of
the named
addressee(s). If you are not the intended recipient, please contact
immediately the
sender by telephone, fax or e-mail and delete the information in this
message that has
been received in error. The sender does not give any warranty or accept
liability as the
content, accuracy or completeness of sent messages and accepts no
responsibility for
changes made after they were sent or for other risks which arise as a
result of e-mail
transmission, viruses, etc.
--
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the experts!
Visithttp://goo.gl/it488V for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer
GeoSolutions S.A.S.
Via di Montramito 3/A 55054 Massarosa
<https://maps.google.com/?q=Via+di+Montramito+3/A%0A55054++Massarosa&entry=gmail&source=g>
(LU)
Italy
phone:+39 0584 962313 <tel:+39%200584%20962313>
fax:+39 0584 1660272 <tel:+39%200584%20166%200272>
http://www.geo-solutions.it
http://twitter.com/geosolutions_it <http://twitter.com/geosolutions_it>
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro
utilizzo è consentito esclusivamente al destinatario del messaggio, per le
finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio
senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal
Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!
http://sdm.link/slashdot_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
<mailto:GeoTools-Devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/geotools-devel
<https://lists.sourceforge.net/lists/listinfo/geotools-devel>
--
--
Jody Garnett
--
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the experts! Visithttp://goo.gl/it488V
for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer
GeoSolutions S.A.S.
Via di Montramito 3/A 55054 Massarosa
<https://maps.google.com/?q=Via+di+Montramito+3/A%0A55054++Massarosa&entry=gmail&source=g>
(LU)
Italy
phone:+39 0584 962313 <tel:+39%200584%20962313>
fax:+39 0584 1660272 <tel:+39%200584%20166%200272>
http://www.geo-solutions.it
http://twitter.com/geosolutions_it <http://twitter.com/geosolutions_it>
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro
utilizzo è consentito esclusivamente al destinatario del messaggio, per le
finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio
senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via
e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal
Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
--
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo
è consentito esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro
sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
