Andrei, I'm not an ES user so I don't fully understand this issue, but my two cents anyway...
Can you show how those examples affect SQL against the ES adapter and/or how they affect JSON models? You seem to be using '_' as a separator character. Are we sure that people will never use it in index or type name? Separator characters often cause problems. Julian On Fri, Jun 29, 2018 at 10:58 AM, Andrei Sereda <[email protected]> wrote: > I agree there should be a configuration option. How about the following > approach. > > Expose both variables ${index} and ${type} in configuration (JSON) and user > will use them to generate table name in calcite schema. > > Example > "table_name": "${type}" // current > "table_name": "${index}" // new (default?) > "table_name": "${index}_${type}" // most generic. supports multiple types > per index > > > > > > On Fri, Jun 29, 2018 at 9:26 AM Michael Mior <[email protected]> wrote: > >> I think it sounds like you and Andrei are in a good position to tackle this >> one so I'm happy to have you both work on whatever solution you think is >> best. >> >> -- >> Michael Mior >> [email protected] >> >> >> >> Le ven. 29 juin 2018 à 04:19, Christian Beikov <[email protected] >> > >> a écrit : >> >> > IMO the best solution would be to make it configurable by introducing a >> > "table_mapping" config with values >> > >> > * type - every type in the known indices is mapped as table >> > * index - every known index is mapped as table >> > >> > We'd probably also need a "type_field" configuration for defining which >> > field to use for the type determination as one of the possible future >> > ways to do things is to introduce a custom field: >> > >> > >> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_custom_type_field_2 >> > >> > We already detect the ES version, so we can set a smart default for this >> > setting. Let's make the index config param optional. >> > >> > * When no index is given, we discover indexes, the default for >> > "table_mapping" then is "index" >> > * When index is given, the we only discover types according to the >> > "type_field" configuration and the default for "table_mapping" is >> > "type" >> > >> > This would also allow to discover indexes but still use "type" as >> > "table_mapping". >> > >> > What do you think? >> > >> > Mit freundlichen Grüßen, >> > ------------------------------------------------------------------------ >> > *Christian Beikov* >> > Am 29.06.2018 um 02:41 schrieb Andrei Sereda: >> > > Yes. There is an API to list all indexes / types in elastic. They can >> be >> > > automatically imported into a schema. >> > > >> > > What needs to be agreed upon is how to expose those elements in calcite >> > > schema (naming / behaviour). >> > > >> > > 1) Many (most?) of setups are single type per index. Natural way to >> name >> > > would be "elastic.$index" (elastic being schema name). Multiple >> indexes >> > > would be under same schema "elastic.index1" "elastic.index2" etc. >> > > >> > > 2) What if index has several types should they exported as calcite >> > tables: >> > > "elastic.$index_type1" "elastic.$index_type2" ? Or (current behaviour) >> > as >> > > "elastic.type1" and "elastic.type2". Or as subschema >> > > "elastic.$index.type1" ? >> > > >> > > Now what if one has combination of (1) and (2) ? >> > > Setup (2) is already deprecated (and will be unsupported in next >> version) >> > > >> > > >> > > On Thu, Jun 28, 2018 at 7:31 PM Christian Beikov < >> > [email protected]> >> > > wrote: >> > > >> > >> Is there an API to discover indexes? If there is, I'd suggest we >> allow a >> > >> config option that to make the adapter discover the possible indexes. >> > >> We'd still have to adapt the code a bit, but internally, the schema >> > >> could just keep a cache of type name to index name map and be able to >> > >> support both scenarios. >> > >> >> > >> >> > >> Mit freundlichen Grüßen, >> > >> >> ------------------------------------------------------------------------ >> > >> *Christian Beikov* >> > >> Am 29.06.2018 um 00:12 schrieb Andrei Sereda: >> > >>>> 1) What's the time horizon for the current adapter no longer working >> > >> with these >> > >>> changes to ES ? >> > >>> Current adapter will be working for a while with existing setup. The >> > >>> problem is nomenclature and ease of use. >> > >>> >> > >>> Their new SQL concepts mapping >> > >>> < >> > >> >> > >> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html >> > >>> drops >> > >>> the notion of ES type (which before was equivalent of RDBMS table) >> and >> > >> uses >> > >>> ES index as new table equivalent (before ES index was equal to >> > database). >> > >>> Most users use elastic this way (one type , one index) index == >> table. >> > >>> >> > >>> Currently calcite requires schema per index. In RDBMS parlance >> database >> > >> per >> > >>> table (I'd like to change that). >> > >>> >> > >>>> 2) Any guess how complicated it would be to maintain code paths for >> > both >> > >>>> behaviours? I know this is probably really challenging to estimate, >> > but >> > >> I >> > >>>> really have no idea of the scope of these changes. Would it mean two >> > >>>> different ES adapters? >> > >>> One can have just a separate calcite schema implementations (same >> > >> adapter / >> > >>> module) : >> > >>> 1) LegacySchema (old). Schema can have only one index (but multiple >> > >>> types). Type == table in this case. >> > >>> 2) NewSchema (new). Single schema can have multiple indexes (type is >> > >>> dropped). Index == table in this case >> > >>> >> > >>>> 3) Do we really need compatibility with the current version of the >> > >>> adapter? >> > >>>> IMO this depends on what versions of ES we would lose support for >> and >> > >> how >> > >>>> complex it would be for users of the current ES adapter to make >> > updates >> > >>> for >> > >>>> any Calcite API changes. >> > >>> The issue is not in adapter but how calcite schema exposes tables. >> > >> Should >> > >>> it expose index as individual table (new), or ES type (old) ? >> > >>> >> > >>> Andrei. >> > >>> >> > >>> On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <[email protected]> >> wrote: >> > >>> >> > >>>> Unfortunately I know very little about ES so I'm not in a great >> > >> position to >> > >>>> asses the impact of these changes. I will say that that legacy >> > >>>> compatibility is great, but maintaining two sets of logic is always >> a >> > >>>> challenge. A few follow up questions: >> > >>>> >> > >>>> 1) What's the time horizon for the current adapter no longer working >> > >> with >> > >>>> these changes to ES? >> > >>>> >> > >>>> 2) Any guess how complicated it would be to maintain code paths for >> > both >> > >>>> behaviours? I know this is probably really challenging to estimate, >> > but >> > >> I >> > >>>> really have no idea of the scope of these changes. Would it mean two >> > >>>> different ES adapters? >> > >>>> >> > >>>> 3) Do we really need compatibility with the current version of the >> > >> adapter? >> > >>>> IMO this depends on what versions of ES we would lose support for >> and >> > >> how >> > >>>> complex it would be for users of the current ES adapter to make >> > updates >> > >> for >> > >>>> any Calcite API changes. >> > >>>> >> > >>>> Thanks for your continued work on the ES adapter Andrei! >> > >>>> >> > >>>> -- >> > >>>> Michael Mior >> > >>>> [email protected] >> > >>>> >> > >>>> >> > >>>> >> > >>>> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <[email protected]> a >> > écrit >> > >> : >> > >>>>> Hello, >> > >>>>> >> > >>>>> Elastic announced >> > >>>>> < >> > >>>>> >> > >> >> > >> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html >> > >>>>> that they will be deprecating mapping types in ES6 and indexes will >> > be >> > >>>>> single-typed only. >> > >>>>> >> > >>>>> Historical analogy <https://www.elastic.co/blog/index-vs-type> >> > between >> > >>>>> RDBMS and elastic was that index is equivalent to a database and >> type >> > >>>>> corresponds to table in that database. In a couple of releases >> > (ES6-8) >> > >>>> this >> > >>>>> shall not longer be true. >> > >>>>> >> > >>>>> Recent SQL addition >> > >>>>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to >> > elastic >> > >>>>> confirms >> > >>>>> this trend >> > >>>>> < >> > >>>>> >> > >> >> > >> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html >> > >>>>>> . >> > >>>>> Index is equivalent to a table and there are no more ES types. >> > >>>>> >> > >>>>> I would like to propose to include this logic in Calcite ES >> adapter. >> > >> IE, >> > >>>>> expose each ES single-typed index as a separate table inside >> calcite >> > >>>>> schema. This is in contrast to current integration where schema >> can >> > >> only >> > >>>>> have a single index. Current approach forces you to create multiple >> > >>>> schemas >> > >>>>> to query single-typed indexes (on the same ES cluster). >> > >>>>> >> > >>>>> Legacy compatibility can always be controlled with configuration >> > >>>>> parameters. >> > >>>>> >> > >>>>> Do you agree with such changes ? If yes, would you consider a PR ? >> > >>>>> >> > >>>>> Regards, >> > >>>>> Andrei. >> > >>>>> >> > >> >> > >> > >>
