I think it sounds like you and Andrei are in a good position to tackle this one so I'm happy to have you both work on whatever solution you think is best.
-- Michael Mior [email protected] Le ven. 29 juin 2018 à 04:19, Christian Beikov <[email protected]> a écrit : > IMO the best solution would be to make it configurable by introducing a > "table_mapping" config with values > > * type - every type in the known indices is mapped as table > * index - every known index is mapped as table > > We'd probably also need a "type_field" configuration for defining which > field to use for the type determination as one of the possible future > ways to do things is to introduce a custom field: > > https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_custom_type_field_2 > > We already detect the ES version, so we can set a smart default for this > setting. Let's make the index config param optional. > > * When no index is given, we discover indexes, the default for > "table_mapping" then is "index" > * When index is given, the we only discover types according to the > "type_field" configuration and the default for "table_mapping" is > "type" > > This would also allow to discover indexes but still use "type" as > "table_mapping". > > What do you think? > > Mit freundlichen Grüßen, > ------------------------------------------------------------------------ > *Christian Beikov* > Am 29.06.2018 um 02:41 schrieb Andrei Sereda: > > Yes. There is an API to list all indexes / types in elastic. They can be > > automatically imported into a schema. > > > > What needs to be agreed upon is how to expose those elements in calcite > > schema (naming / behaviour). > > > > 1) Many (most?) of setups are single type per index. Natural way to name > > would be "elastic.$index" (elastic being schema name). Multiple indexes > > would be under same schema "elastic.index1" "elastic.index2" etc. > > > > 2) What if index has several types should they exported as calcite > tables: > > "elastic.$index_type1" "elastic.$index_type2" ? Or (current behaviour) > as > > "elastic.type1" and "elastic.type2". Or as subschema > > "elastic.$index.type1" ? > > > > Now what if one has combination of (1) and (2) ? > > Setup (2) is already deprecated (and will be unsupported in next version) > > > > > > On Thu, Jun 28, 2018 at 7:31 PM Christian Beikov < > [email protected]> > > wrote: > > > >> Is there an API to discover indexes? If there is, I'd suggest we allow a > >> config option that to make the adapter discover the possible indexes. > >> We'd still have to adapt the code a bit, but internally, the schema > >> could just keep a cache of type name to index name map and be able to > >> support both scenarios. > >> > >> > >> Mit freundlichen Grüßen, > >> ------------------------------------------------------------------------ > >> *Christian Beikov* > >> Am 29.06.2018 um 00:12 schrieb Andrei Sereda: > >>>> 1) What's the time horizon for the current adapter no longer working > >> with these > >>> changes to ES ? > >>> Current adapter will be working for a while with existing setup. The > >>> problem is nomenclature and ease of use. > >>> > >>> Their new SQL concepts mapping > >>> < > >> > https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html > >>> drops > >>> the notion of ES type (which before was equivalent of RDBMS table) and > >> uses > >>> ES index as new table equivalent (before ES index was equal to > database). > >>> Most users use elastic this way (one type , one index) index == table. > >>> > >>> Currently calcite requires schema per index. In RDBMS parlance database > >> per > >>> table (I'd like to change that). > >>> > >>>> 2) Any guess how complicated it would be to maintain code paths for > both > >>>> behaviours? I know this is probably really challenging to estimate, > but > >> I > >>>> really have no idea of the scope of these changes. Would it mean two > >>>> different ES adapters? > >>> One can have just a separate calcite schema implementations (same > >> adapter / > >>> module) : > >>> 1) LegacySchema (old). Schema can have only one index (but multiple > >>> types). Type == table in this case. > >>> 2) NewSchema (new). Single schema can have multiple indexes (type is > >>> dropped). Index == table in this case > >>> > >>>> 3) Do we really need compatibility with the current version of the > >>> adapter? > >>>> IMO this depends on what versions of ES we would lose support for and > >> how > >>>> complex it would be for users of the current ES adapter to make > updates > >>> for > >>>> any Calcite API changes. > >>> The issue is not in adapter but how calcite schema exposes tables. > >> Should > >>> it expose index as individual table (new), or ES type (old) ? > >>> > >>> Andrei. > >>> > >>> On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <[email protected]> wrote: > >>> > >>>> Unfortunately I know very little about ES so I'm not in a great > >> position to > >>>> asses the impact of these changes. I will say that that legacy > >>>> compatibility is great, but maintaining two sets of logic is always a > >>>> challenge. A few follow up questions: > >>>> > >>>> 1) What's the time horizon for the current adapter no longer working > >> with > >>>> these changes to ES? > >>>> > >>>> 2) Any guess how complicated it would be to maintain code paths for > both > >>>> behaviours? I know this is probably really challenging to estimate, > but > >> I > >>>> really have no idea of the scope of these changes. Would it mean two > >>>> different ES adapters? > >>>> > >>>> 3) Do we really need compatibility with the current version of the > >> adapter? > >>>> IMO this depends on what versions of ES we would lose support for and > >> how > >>>> complex it would be for users of the current ES adapter to make > updates > >> for > >>>> any Calcite API changes. > >>>> > >>>> Thanks for your continued work on the ES adapter Andrei! > >>>> > >>>> -- > >>>> Michael Mior > >>>> [email protected] > >>>> > >>>> > >>>> > >>>> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <[email protected]> a > écrit > >> : > >>>>> Hello, > >>>>> > >>>>> Elastic announced > >>>>> < > >>>>> > >> > https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html > >>>>> that they will be deprecating mapping types in ES6 and indexes will > be > >>>>> single-typed only. > >>>>> > >>>>> Historical analogy <https://www.elastic.co/blog/index-vs-type> > between > >>>>> RDBMS and elastic was that index is equivalent to a database and type > >>>>> corresponds to table in that database. In a couple of releases > (ES6-8) > >>>> this > >>>>> shall not longer be true. > >>>>> > >>>>> Recent SQL addition > >>>>> <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to > elastic > >>>>> confirms > >>>>> this trend > >>>>> < > >>>>> > >> > https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html > >>>>>> . > >>>>> Index is equivalent to a table and there are no more ES types. > >>>>> > >>>>> I would like to propose to include this logic in Calcite ES adapter. > >> IE, > >>>>> expose each ES single-typed index as a separate table inside calcite > >>>>> schema. This is in contrast to current integration where schema can > >> only > >>>>> have a single index. Current approach forces you to create multiple > >>>> schemas > >>>>> to query single-typed indexes (on the same ES cluster). > >>>>> > >>>>> Legacy compatibility can always be controlled with configuration > >>>>> parameters. > >>>>> > >>>>> Do you agree with such changes ? If yes, would you consider a PR ? > >>>>> > >>>>> Regards, > >>>>> Andrei. > >>>>> > >> > >
