[
https://issues.apache.org/jira/browse/GRIFFIN-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XIAOYU YU updated GRIFFIN-322:
------------------------------
Description:
The ElasticSearch connector was added in Griffin. We want to enrich it by
adding sql mode.
ES SQL plugins are very popular, such as :
[https://github.com/NLPchina/elasticsearch-sql/wiki
|https://github.com/NLPchina/elasticsearch-sql/wiki]and
[https://www.elastic.co/guide/en/elasticsearch/reference/7.x/xpack-sql.html].
For example:
{code:java}
curl -X POST 'http://{es_url}/_sql?format=csv' -H 'Content-Type:
application/json' -d 'select a, b, c from {index} where a is not null limit
2'{code}
would get csv response as follows:
{code:java}
a,b,c
1,test,1.55
2,test,1.57{code}
So we want Griffin ES Connector could load data by ES SQL in sql mode. In sql
mode, user could config Connector for ES (with sql plugin) like:
_{ "connectors": [{_
_"type": "custom",_
_"config": {_
_"class":
"{{org.apache.griffin.measure.datasource.connector.batch.ElasticSearchGriffinDataConnector}}",_
_"host": "xxxx", "port": xxxx, "sql": "select a, b, c from xxx where a is not
null" }_
_]}_
Which has the following benefits:
# For case that only part of the ES data is needed, or only aggregated result
is needed, it could significantly reduce IO and improve performance by handle
filter and aggregation to ES.
# It is more user-friendly, since we don't need to define data size and
schema.
was:
The ElasticSearch connector was added in Griffin. We want to enrich it by
adding sql mode.
ES SQL plugins are very popular, such as :
[https://github.com/NLPchina/elasticsearch-sql/wiki
|https://github.com/NLPchina/elasticsearch-sql/wiki]and
[https://www.elastic.co/guide/en/elasticsearch/reference/7.x/xpack-sql.html].
For example:
{code:java}
curl -X POST 'http://{es_url}/_sql?format=csv' -H 'Content-Type:
application/json' -d 'select a, b, c from {index} where a is not null limit
2'{code}
would get csv response as follows:
{code:java}
a,b,c
1,test,1.55
2,test,1.57{code}
So we want Griffin ES Connector could load data by ES SQL in sql mode. In sql
mode, user could config Connector for ES (with sql plugin) like:
_{ "connectors": [{_
_"type": "ElasticSearch",_
_"config": \{ "host": "xxxx", "port": xxxx, "sql": "select a, b, c from xxx
where a is not null" }_
_]}_
Which has the following benefits:
# For case that only part of the ES data is needed, or only aggregated result
is needed, it could significantly reduce IO and improve performance by handle
filter and aggregation to ES.
# It is more user-friendly, since we don't need to define data size and
schema.
> Add SQL mode for ES connector
> -----------------------------
>
> Key: GRIFFIN-322
> URL: https://issues.apache.org/jira/browse/GRIFFIN-322
> Project: Griffin
> Issue Type: Improvement
> Reporter: XIAOYU YU
> Priority: Major
>
> The ElasticSearch connector was added in Griffin. We want to enrich it by
> adding sql mode.
> ES SQL plugins are very popular, such as :
> [https://github.com/NLPchina/elasticsearch-sql/wiki
> |https://github.com/NLPchina/elasticsearch-sql/wiki]and
> [https://www.elastic.co/guide/en/elasticsearch/reference/7.x/xpack-sql.html].
> For example:
> {code:java}
> curl -X POST 'http://{es_url}/_sql?format=csv' -H 'Content-Type:
> application/json' -d 'select a, b, c from {index} where a is not null limit
> 2'{code}
> would get csv response as follows:
>
> {code:java}
> a,b,c
> 1,test,1.55
> 2,test,1.57{code}
>
> So we want Griffin ES Connector could load data by ES SQL in sql mode. In
> sql mode, user could config Connector for ES (with sql plugin) like:
>
> _{ "connectors": [{_
> _"type": "custom",_
> _"config": {_
> _"class":
> "{{org.apache.griffin.measure.datasource.connector.batch.ElasticSearchGriffinDataConnector}}",_
> _"host": "xxxx", "port": xxxx, "sql": "select a, b, c from xxx where a is not
> null" }_
> _]}_
>
> Which has the following benefits:
> # For case that only part of the ES data is needed, or only aggregated
> result is needed, it could significantly reduce IO and improve performance by
> handle filter and aggregation to ES.
> # It is more user-friendly, since we don't need to define data size and
> schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
