wuyunfeng opened a new issue #701: Support querying Elasticserch, Using Doris and Elasticsearch for real-time data analysis URL: https://github.com/apache/incubator-doris/issues/701 **Is your feature request related to a problem? Please describe.** Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases, but do not support join or distribute join on two or more index. Doris not only support this feature, but aslo providing QEP(Query Execution Plan) thanks for it's built in Query Engine **Describe the solution you'd like** In order to support querying elasticsearch, we can implement an `elasticsearch scan node` for doris **Describe alternatives you've considered** **Additional context** 1. create external table for elasticsearch index, such as ` CREATE EXTERNAL TABLE `es_table` ( `id` bigint(20) COMMENT "", `k1` bigint(20) COMMENT "", `k2` datetime COMMENT "", `k3` varchar(20) COMMENT "", `k4` varchar(100) COMMENT "", `k5` float COMMENT "" ) ENGINE=ELASTICSEARCH PARTITION BY RANGE(`id`) () PROPERTIES ( "host" = "host", "user" = "xxxx", "password" = "xxxx", "index" = "tindex”, "type" = "doc" ); ` 2. then you can using the plain SQL query on the table, or doing some complex join operation etc. such as ` select * from es_table where k1 > 1000 and k3 ='term' or k4 like 'fu*z_' ` We support pushdown filters, this significantly increases the queries performance and minimizes the CPU, memory and I/O on both doris and Elasticsearch clusters as only the needed data is returned. In addition to this, you can using the Elasticsearch QueryDSL with the help of `esquery` function(we supplement the sql syntax for external Elasticsearch table): ` select * from es_table where esquery('k4', '{ "match": { "k4": "doris on elasticsearch" } }'); ` Operators those have been optimized as pushdown filters: | SQL syntax | ES 5.x+ syntax | |-------|:---:| | = | term query| | in | terms query | | > , < , >= , ⇐ | range | | and | bool.filter | | or | bool.should | | not | bool.must_not | | not in | bool.must_not + terms | | esquery | basic dsl |
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
