Dear all,
sorry for the cross posting on both mailing list ... but well App-Schema "as a
leg" on both projects
and this may be interesting for people on both mailing lists.
... and my apologies for the wall of text that follows ... what can I say
complex features are really
a complex subject :)
Lately I have been working in integrating App-Schema with Apache Solr [1] which
is a well know
solution for indexing and (faceting) searching document.
We already have an Apache Solr store in both GT and GS, configuring a layer for
it requires the user
to provide some extra information, e.g. which index fields should be
considered, should multivalued
fields be used, should empty fields be used and which is attribute should be
considered the
default geometry.
The first step for the integration between Apache Solr and App-Schema is to
allow App-Schema to
use Apache Solr has it one of its data store. When in the context of App-schema
we already have
most of the information we need, we already say indirectly through the mappings
which fields of
the index we are interested in, which ones are multivalued (<multivalue>), etc
...
The Apache-Solr data store configuration doesn't fit in the usual MAP
parameters configuration
model ... and we also need to automatically detect during the parsing of the
data stores
configuration that we are dealing with an Apache Solr data store and do all the
possible
automatic steps.
I have come up with the following syntax for the Apache Sorl data store
configuration, please find
the complete mappings file attached to this mail (mappings_solr.xml):
(...)
<sourceDataStores>
<SolrDataStore>
<id>stations</id>
<url>http://localhost:8983/solr/stations</url>
<index name="stations">
<geometry default="true">
<name>station_location</name>
<srid>4326</srid>
<type>POINT</type>
</geometry>
</index>
</SolrDataStore>
</sourceDataStores>
(...)
All the other necessary information will be automatically inferred from the
App-Schema mappings
file when possible, at the exception of multivalued fields. An Apache Solr
multivalued field
handles \ contains multiple values for the same attribute.
Apache Solr multivalued fields are related with App-Schema multivalued fields,
i.e. simple attributes
(String, numeric, dates, etc ...) that can be encoded multiple times. Currently
in App-Schema
this attributes can be mapped using a denormalized data store (database) or
using a specific
feature chaining syntax when client properties are involved.
Here I would like to extend the App-Schema notion of multivalued attributes and
allow each data
data store to contribute a specific handler with a custom syntax has needed:
* Apache Solr could use multivalued attributes
* MongoDB could use JSON arrays
* SQL could define explicit foreign key relation
Please find full examples of mappings using custom handlers attached to this
mail (multi_*). For SQL
data stores the syntax looks like this:
(...)
<AttributeMapping>
<targetAttribute>st:comment</targetAttribute>
<jdbcMultipleValue>
<sourceColumn>id</sourceColumn>
<targetTable>meteo_stations_comments</targetTable>
<targetColumn>station_id</targetColumn>
<targetValue>comment</targetValue>
</jdbcMultipleValue>
</AttributeMapping>
(...)
A station can have multiple comment associated to it, this comment are stored in
/meteo_stations_comments
/table which has a foreign key to /stations /table base on columns /id /and /station_id/. The target
value can be an
OCQL expression.
//
In the case of Apache Solr the syntax would look like this:
(...)
<AttributeMapping>
<targetAttribute>st:comment</targetAttribute>
<solrMultipleValue>station_comment</solrMultipleValue>
</AttributeMapping>
(...)
I have a prototype of all of this already working, all of the things above required creating a few
extensions
points in App-Schema:
* in the mappings configuration parsing module to handle the custom syntaxes
of each data store
* in the App-Schema feature iterator to give a chance to custom handlers to
do their job
This changes open the doors to the usage of non relational data stores with App-Schema and IMHO this
syntaxes
promote \ allow clearer mappings files, of curse this changes and new syntaxes will need to be
clearly documented
in the App-Schema documentation.
All comments about this are welcome.
Kind regards,
Nuno Oliveira
--
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i
file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo
è consentito esclusivamente al destinatario del messaggio, per le finalità
indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne
il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di
procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro
sistema. Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse,
costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<as:AppSchemaDataAccess xmlns:as="http://www.geotools.org/app-schema"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
<namespaces>
<Namespace>
<prefix>st</prefix>
<uri>http://www.stations.org/1.0</uri>
</Namespace>
<Namespace>
<prefix>gml</prefix>
<uri>http://www.opengis.net/gml/3.2</uri>
</Namespace>
</namespaces>
<sourceDataStores>
<SolrDataStore>
<id>stations</id>
<url>http://localhost:8983/solr/stations</url>
<index name="stations">
<geometry default="true">
<name>station_location</name>
<srid>4326</srid>
<type>POINT</type>
</geometry>
</index>
</SolrDataStore>
</sourceDataStores>
<targetTypes>
<FeatureType>
<schemaUri>meteo.xsd</schemaUri>
</FeatureType>
</targetTypes>
<typeMappings>
<FeatureTypeMapping>
<mappingName>stations_solr</mappingName>
<sourceDataStore>stations</sourceDataStore>
<sourceType>stations</sourceType>
<targetElement>st:Station</targetElement>
<attributeMappings>
<AttributeMapping>
<targetAttribute>st:Station</targetAttribute>
<idExpression>
<OCQL>station_id</OCQL>
</idExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:stationName</targetAttribute>
<sourceExpression>
<OCQL>station_name</OCQL>
</sourceExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:position</targetAttribute>
<sourceExpression>
<OCQL>station_location</OCQL>
</sourceExpression>
</AttributeMapping>
</attributeMappings>
</FeatureTypeMapping>
</typeMappings>
</as:AppSchemaDataAccess><?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<as:AppSchemaDataAccess xmlns:as="http://www.geotools.org/app-schema"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
<namespaces>
<Namespace>
<prefix>st</prefix>
<uri>http://www.stations.org/1.0</uri>
</Namespace>
<Namespace>
<prefix>gml</prefix>
<uri>http://www.opengis.net/gml/3.2</uri>
</Namespace>
</namespaces>
<sourceDataStores>
<SolrDataStore>
<id>stations</id>
<url>http://localhost:8983/solr/stations</url>
<index name="stations">
<geometry default="true">
<name>station_location</name>
<srid>4326</srid>
<type>POINT</type>
</geometry>
</index>
</SolrDataStore>
</sourceDataStores>
<targetTypes>
<FeatureType>
<schemaUri>meteo.xsd</schemaUri>
</FeatureType>
</targetTypes>
<typeMappings>
<FeatureTypeMapping>
<mappingName>multi_stations_solr</mappingName>
<sourceDataStore>stations</sourceDataStore>
<sourceType>stations</sourceType>
<targetElement>st:Station</targetElement>
<attributeMappings>
<AttributeMapping>
<targetAttribute>st:Station</targetAttribute>
<idExpression>
<OCQL>station_id</OCQL>
</idExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:stationName</targetAttribute>
<sourceExpression>
<OCQL>station_name</OCQL>
</sourceExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:position</targetAttribute>
<sourceExpression>
<OCQL>station_location</OCQL>
</sourceExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:tag</targetAttribute>
<solrMultipleValue>station_tag</solrMultipleValue>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:comment</targetAttribute>
<solrMultipleValue>station_comment</solrMultipleValue>
</AttributeMapping>
</attributeMappings>
</FeatureTypeMapping>
</typeMappings>
</as:AppSchemaDataAccess><?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns3:AppSchemaDataAccess xmlns:ns2="http://www.opengis.net/ogc"; xmlns:ns3="http://www.geotools.org/app-schema";>
<namespaces>
<Namespace>
<prefix>st</prefix>
<uri>http://www.stations.org/1.0</uri>
</Namespace>
<Namespace>
<prefix>gml</prefix>
<uri>http://www.opengis.net/gml/3.2</uri>
</Namespace>
</namespaces>
<sourceDataStores>
<DataStore>
<id>dataStore</id>
<parameters>
***
</parameters>
</DataStore>
</sourceDataStores>
<targetTypes>
<FeatureType>
<schemaUri>meteo.xsd</schemaUri>
</FeatureType>
</targetTypes>
<typeMappings>
<FeatureTypeMapping>
<mappingName>multi_stations_postgres</mappingName>
<sourceDataStore>dataStore</sourceDataStore>
<sourceType>meteo_stations</sourceType>
<targetElement>st:Station</targetElement>
<attributeMappings>
<AttributeMapping>
<targetAttribute>st:Station</targetAttribute>
<idExpression>
<OCQL>id</OCQL>
</idExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:stationName</targetAttribute>
<sourceExpression>
<OCQL>common_name</OCQL>
</sourceExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:position</targetAttribute>
<sourceExpression>
<OCQL>position</OCQL>
</sourceExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:tag</targetAttribute>
<jdbcMultipleValue>
<sourceColumn>id</sourceColumn>
<targetTable>meteo_stations_tags</targetTable>
<targetColumn>station_id</targetColumn>
<targetValue>tag</targetValue>
</jdbcMultipleValue>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:comment</targetAttribute>
<jdbcMultipleValue>
<sourceColumn>id</sourceColumn>
<targetTable>meteo_stations_comments</targetTable>
<targetColumn>station_id</targetColumn>
<targetValue>comment</targetValue>
</jdbcMultipleValue>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:observation</targetAttribute>
<sourceExpression>
<OCQL>id</OCQL>
<linkElement>multi_observations_postgres</linkElement>
<linkField>FEATURE_LINK[1]</linkField>
</sourceExpression>
<isMultiple>true</isMultiple>
</AttributeMapping>
</attributeMappings>
</FeatureTypeMapping>
<FeatureTypeMapping>
<mappingName>multi_observations_postgres</mappingName>
<sourceDataStore>dataStore</sourceDataStore>
<sourceType>meteo_observations</sourceType>
<targetElement>st:Observation</targetElement>
<attributeMappings>
<AttributeMapping>
<targetAttribute>st:Observation</targetAttribute>
<idExpression>
<OCQL>id</OCQL>
</idExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:value</targetAttribute>
<sourceExpression>
<OCQL>value</OCQL>
</sourceExpression>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:tag</targetAttribute>
<jdbcMultipleValue>
<sourceColumn>id</sourceColumn>
<targetTable>meteo_observations_tags</targetTable>
<targetColumn>observation_id</targetColumn>
<targetValue>tag</targetValue>
</jdbcMultipleValue>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>st:comment</targetAttribute>
<jdbcMultipleValue>
<sourceColumn>id</sourceColumn>
<targetTable>meteo_observations_comments</targetTable>
<targetColumn>observation_id</targetColumn>
<targetValue>comment</targetValue>
</jdbcMultipleValue>
</AttributeMapping>
<AttributeMapping>
<targetAttribute>FEATURE_LINK[1]</targetAttribute>
<sourceExpression>
<OCQL>station_id</OCQL>
</sourceExpression>
</AttributeMapping>
</attributeMappings>
</FeatureTypeMapping>
</typeMappings>
</ns3:AppSchemaDataAccess>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
