Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

I like the idea of the regex filter, might be cool to have something like that in general for all adapters, but it's fine if you do it just for ES now. I guess you are considering include and exclude pattern parameters? I'm more for a mode parameter and not let the user decide the name

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Christian / Michael, Can you please weight-in for your preferred solution and I'll implement it. One more question. Sometimes it is nice to be able to filter (limit) indexes (tables) exposed by calcite. Say my cluster has 10 indexes but I want user to query only one. Would you be opposed if I

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

That's a reasonable alternative. On Fri, Jun 29, 2018 at 7:57 PM Julian Hyde wrote: > Maybe there could be a separator char as one of the adapter’s parameters. > People should choose a value, say ‘$’ or ‘#’, that is legal in an unquoted > SQL identifier but does not occur in any of their index

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Maybe there could be a separator char as one of the adapter’s parameters. People should choose a value, say ‘$’ or ‘#’, that is legal in an unquoted SQL identifier but does not occur in any of their index or type names. If not specified, the adapter would end up in a simple mode, say looking

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

That's a valid point. Then user would define a different pattern like "i$index_t$type" for his cluster. I think we should first answer wherever such scenarios should be supported by calcite (given that they're already deprecated by the vendor). If yes, what should be collision strategy ? User

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

> In elastic (index/type) pair is guaranteed to be unique therefore > "${index}_${type}" will be also unique (as string). This is only necessary > when we have several types per index. Valid question is wherever user > should be allowed such flexibility. Uniqueness is not my concern. Suppose

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Let's assume we have the following indexes / types in ES (same cluster) (Index with two types. Legacy format. Not supported in ES8+) - index1: t1 + t2 (Index with single type. Default and most used format) - index2: t1 (Index with two types having same name as index1) - index3: t1 + t2

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

I'm not sure what the benefit of allowing users to specify this scheme would be. We'd have to parse it, interpret it, make sure the expressions don't result conflicting names etc. IMO a simple mode configuration would be way easier to implement and probably cover 99% of the use cases. Mit

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Andrei, I'm not an ES user so I don't fully understand this issue, but my two cents anyway... Can you show how those examples affect SQL against the ES adapter and/or how they affect JSON models? You seem to be using '_' as a separator character. Are we sure that people will never use it in

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Plus allow to filter indexes using regexp ? On Fri, Jun 29, 2018 at 1:58 PM Andrei Sereda wrote: > I agree there should be a configuration option. How about the following > approach. > > Expose both variables ${index} and ${type} in configuration (JSON) and > user will use them to generate

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

I agree there should be a configuration option. How about the following approach. Expose both variables ${index} and ${type} in configuration (JSON) and user will use them to generate table name in calcite schema. Example "table_name": "${type}" // current "table_name": "${index}" // new

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

I think it sounds like you and Andrei are in a good position to tackle this one so I'm happy to have you both work on whatever solution you think is best. -- Michael Mior mm...@apache.org Le ven. 29 juin 2018 à 04:19, Christian Beikov a écrit : > IMO the best solution would be to make it

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

IMO the best solution would be to make it configurable by introducing a "table_mapping" config with values * type - every type in the known indices is mapped as table * index - every known index is mapped as table We'd probably also need a "type_field" configuration for defining which field

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Yes. There is an API to list all indexes / types in elastic. They can be automatically imported into a schema. What needs to be agreed upon is how to expose those elements in calcite schema (naming / behaviour). 1) Many (most?) of setups are single type per index. Natural way to name would be

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Is there an API to discover indexes? If there is, I'd suggest we allow a config option that to make the adapter discover the possible indexes. We'd still have to adapt the code a bit, but internally, the schema could just keep a cache of type name to index name map and be able to support both

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

> 1) What's the time horizon for the current adapter no longer working with > these changes to ES ? Current adapter will be working for a while with existing setup. The problem is nomenclature and ease of use. Their new SQL concepts mapping

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Unfortunately I know very little about ES so I'm not in a great position to asses the impact of these changes. I will say that that legacy compatibility is great, but maintaining two sets of logic is always a challenge. A few follow up questions: 1) What's the time horizon for the current adapter

Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table

Hello, Elastic announced that they will be deprecating mapping types in ES6 and indexes will be single-typed only. Historical analogy between RDBMS and elastic was