I have also checked "mvn javadoc:javadoc javadoc:test-javadoc" on my machine (Ubuntu, JDK 8, Maven 3.3.9) and it fails on elasticsearch2 module (didn't check this problem for the next modules).
"mvn clean -DskipTests site" fails for MongoDB module. Kind regards, Volodymyr Vysotskyi сб, 23 черв. 2018 о 03:12 Julian Hyde <[email protected]> пише: > Same failure for me on JDK 8/macOS. The most likely other variable is > maven: I have 3.5.2 on macOS, 3.5.3 on ubuntu. > > I’m mystified why no one else is seeing javadoc failures on macOS. > > Julian > > > > On Jun 22, 2018, at 4:35 PM, Julian Hyde <[email protected]> wrote: > > > > On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following > failure: > > > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > project calcite-mongodb: Error generating > maven-javadoc-plugin:3.0.1:test-javadoc report: > > [ERROR] Exit code: 1 - > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22: > error: cannot find symbol > > [ERROR] import org.apache.calcite.test.CalciteAssert; > > [ERROR] ^ > > [ERROR] symbol: class CalciteAssert > > [ERROR] location: package org.apache.calcite.test > > [ERROR] > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137: > error: package CalciteAssert does not exist > > [ERROR] private CalciteAssert.AssertThat assertModel(String model) { > > [ERROR] ^ > > [ERROR] > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145: > error: package CalciteAssert does not exist > > [ERROR] private CalciteAssert.AssertThat assertModel(URL url) { > > [ERROR] ^ > > [ERROR] > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43: > error: reference not found > > [ERROR] /** Similar to {@link CalciteAssert#checkResultUnordered}, but > filters strings > > [ERROR] ^ > > > > > > > >> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <[email protected]> wrote: > >> > >> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is JDK 8 > vs > >> 9 ? > >> > >> # JDK8 (oracle) / macOS > >> $ mvn clean -DskipTests site > >> > >> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <[email protected]> wrote: > >> > >>> I just tried it again. The following fails for me (jdk9, ubuntu): > >>> > >>> $ git checkout c12cb4b0de > >>> $ mvn clean -DskipTests site > >>> > >>> Constructing Javadoc information... > >>> 1 error > >>> [INFO] > >>> > ------------------------------------------------------------------------ > >>> [INFO] Reactor Summary: > >>> [INFO] > >>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE > [02:29 > >>> min] > >>> [INFO] Calcite Linq4j ..................................... SKIPPED > >>> [INFO] Calcite Core ....................................... SKIPPED > >>> [INFO] Calcite Cassandra .................................. SKIPPED > >>> [INFO] Calcite Druid ...................................... SKIPPED > >>> [INFO] Calcite Elasticsearch .............................. SKIPPED > >>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED > >>> [INFO] Calcite Examples ................................... SKIPPED > >>> [INFO] Calcite Example CSV ................................ SKIPPED > >>> [INFO] Calcite Example Function ........................... SKIPPED > >>> [INFO] Calcite File ....................................... SKIPPED > >>> [INFO] Calcite Geode ...................................... SKIPPED > >>> [INFO] Calcite MongoDB .................................... SKIPPED > >>> [INFO] Calcite Pig ........................................ SKIPPED > >>> [INFO] Calcite Piglet ..................................... SKIPPED > >>> [INFO] Calcite Plus ....................................... SKIPPED > >>> [INFO] Calcite Server ..................................... SKIPPED > >>> [INFO] Calcite Spark ...................................... SKIPPED > >>> [INFO] Calcite Splunk ..................................... SKIPPED > >>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED > >>> [INFO] > >>> > ------------------------------------------------------------------------ > >>> [INFO] BUILD FAILURE > >>> [INFO] > >>> > ------------------------------------------------------------------------ > >>> [INFO] Total time: 02:29 min > >>> [INFO] Finished at: 2018-06-22T14:53:13-07:00 > >>> [INFO] > >>> > ------------------------------------------------------------------------ > >>> [ERROR] Failed to execute goal > >>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > >>> project calcite: Error generating > maven-javadoc-plugin:3.0.1:test-aggregate > >>> report: > >>> [ERROR] Exit code: 1 - > >>> > /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > >>> error: package org.elasticsearch.node.internal does not exist > >>> [ERROR] import > org.elasticsearch.node.internal.InternalSettingsPreparer; > >>> [ERROR] ^ > >>> [ERROR] > >>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options > @packages > >>> @argfile > >>> [ERROR] > >>> [ERROR] Refer to the generated Javadoc files in > >>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir. > >>> [ERROR] -> [Help 1] > >>> [ERROR] > >>> [ERROR] To see the full stack trace of the errors, re-run Maven with > the > >>> -e switch. > >>> [ERROR] Re-run Maven using the -X switch to enable full debug logging. > >>> [ERROR] > >>> [ERROR] For more information about the errors and possible solutions, > >>> please read the following articles: > >>> [ERROR] [Help 1] > >>> > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > >>> > >>> > >>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <[email protected]> wrote: > >>>> > >>>> Odd, mvn clean site still works fine for me. > >>>> -- > >>>> Michael Mior > >>>> [email protected] > >>>> > >>>> > >>>> > >>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <[email protected]> a écrit > : > >>>> > >>>>> Looks like this change broke “mvn site” (perhaps also “mvn > >>>>> javadoc:test-javadoc”). > >>>>> > >>>>> [ERROR] Failed to execute goal > >>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > >>>>> project calcite: Error generating > >>> maven-javadoc-plugin:3.0.1:test-aggregate > >>>>> report: > >>>>> [ERROR] Exit code: 1 - > >>>>> > >>> > /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > >>>>> error: package org.elasticsearch.node.internal does not exist > >>>>> [ERROR] import > org.elasticsearch.node.internal.InternalSettingsPreparer; > >>>>> [ERROR] ^ > >>>>> [ERROR] > >>>>> > >>>>>> On Jun 21, 2018, at 3:39 AM, [email protected] wrote: > >>>>>> > >>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit > tests of > >>>>> ES adapter (Andrei Sereda) > >>>>>> > >>>>>> After discussion on dev-list Integration tests (for ES) have been > >>>>> removed. They're now > >>>>>> superseded by unit tests (which execute queries against a real > elastic > >>>>> instance) > >>>>>> > >>>>>> Added local file (zips-mini.json) which contains a small subset of > >>>>> original zips.json > >>>>>> (allows to bootstrap tests faster) > >>>>>> > >>>>>> Created separate ES JUnit rule which can be re-used across different > >>>>> tests. > >>>>>> > >>>>>> Both v2 and v5 of ES adapters are supported. > >>>>>> > >>>>>> Close apache/calcite#716 > >>>>>> > >>>>>> > >>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo > >>>>>> Commit: > http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0 > >>>>>> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0 > >>>>>> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0 > >>>>>> > >>>>>> Branch: refs/heads/master > >>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d > >>>>>> Parents: 37944bb > >>>>>> Author: Andrei Sereda <[email protected]> > >>>>>> Authored: Thu May 31 18:19:10 2018 -0400 > >>>>>> Committer: Michael Mior <[email protected]> > >>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400 > >>>>>> > >>>>>> > ---------------------------------------------------------------------- > >>>>>> .../AbstractElasticsearchTable.java | 12 + > >>>>>> .../elasticsearch/ElasticsearchProject.java | 61 ++- > >>>>>> elasticsearch2/pom.xml | 6 + > >>>>>> .../Elasticsearch2Enumerator.java | 12 +- > >>>>>> .../elasticsearch2/Elasticsearch2Schema.java | 16 +- > >>>>>> .../elasticsearch2/Elasticsearch2Table.java | 9 +- > >>>>>> .../ElasticSearch2AdapterTest.java | 395 > >>> ++++++++++++++++++ > >>>>>> .../elasticsearch2/EmbeddedElasticNode.java | 147 +++++++ > >>>>>> .../elasticsearch2/EmbeddedElasticRule.java | 97 +++++ > >>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ > >>>>>> .../calcite/test/Elasticsearch2AdapterIT.java | 270 ------------- > >>>>>> .../resources/elasticsearch-zips-model.json | 50 --- > >>>>>> .../src/test/resources/zips-mini.json | 149 +++++++ > >>>>>> elasticsearch5/pom.xml | 31 ++ > >>>>>> .../elasticsearch5/Elasticsearch5Schema.java | 17 +- > >>>>>> .../elasticsearch5/Elasticsearch5Table.java | 11 +- > >>>>>> .../ElasticSearch5AdapterTest.java | 399 > >>> +++++++++++++++++++ > >>>>>> .../elasticsearch5/EmbeddedElasticNode.java | 153 +++++++ > >>>>>> .../elasticsearch5/EmbeddedElasticRule.java | 98 +++++ > >>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ > >>>>>> .../calcite/test/Elasticsearch5AdapterIT.java | 270 ------------- > >>>>>> .../resources/elasticsearch-zips-model.json | 50 --- > >>>>>> elasticsearch5/src/test/resources/log4j2.xml | 16 + > >>>>>> .../src/test/resources/zips-mini.json | 149 +++++++ > >>>>>> pom.xml | 20 +- > >>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-) > >>>>>> > ---------------------------------------------------------------------- > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>> > >>> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>> index 0980469..8cc5933 100644 > >>>>>> --- > >>>>> > >>> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>> +++ > >>>>> > >>> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>>>>> @@ -75,6 +75,18 @@ public abstract class AbstractElasticsearchTable > >>>>> extends AbstractQueryableTable > >>>>>> relOptTable, this, null); > >>>>>> } > >>>>>> > >>>>>> + /** > >>>>>> + * In ES 5.x scripted fields start with {@code > params._source.foo} > >>>>> while in ES2.x > >>>>>> + * {@code _source.foo}. Helper method to build correct query > based > >>> on > >>>>> runtime version of elastic. > >>>>>> + * > >>>>>> + * @see <a href=" > >>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source > >>>>> variable</a> > >>>>>> + * @see <a href=" > >>>>> > >>> > https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html > >>> ">Scripted > >>>>> Fields</a> > >>>>>> + */ > >>>>>> + protected String scriptedFieldPrefix() { > >>>>>> + // this is default pattern starting 5.x > >>>>>> + return "params._source"; > >>>>>> + } > >>>>>> + > >>>>>> /** Executes a "find" operation on the underlying type. > >>>>>> * > >>>>>> * <p>For example, > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>> > >>> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>> index b42abd7..961c8b0 100644 > >>>>>> --- > >>>>> > >>> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>> +++ > >>>>> > >>> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>>>>> @@ -27,11 +27,15 @@ import > >>>>> org.apache.calcite.rel.metadata.RelMetadataQuery; > >>>>>> import org.apache.calcite.rel.type.RelDataType; > >>>>>> import org.apache.calcite.rex.RexNode; > >>>>>> import org.apache.calcite.util.Pair; > >>>>>> -import org.apache.calcite.util.Util; > >>>>>> + > >>>>>> +import com.google.common.base.Function; > >>>>>> +import com.google.common.collect.Lists; > >>>>>> > >>>>>> import java.util.ArrayList; > >>>>>> import java.util.List; > >>>>>> > >>>>>> +import javax.annotation.Nullable; > >>>>>> + > >>>>>> /** > >>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project} > >>>>>> * relational expression in Elasticsearch. > >>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends > Project > >>>>> implements ElasticsearchRel { > >>>>>> implementor.visitChild(0, getInput()); > >>>>>> > >>>>>> final List<String> inFields = > >>>>>> - > >>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); > >>>>>> + > >>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); > >>>>>> final ElasticsearchRules.RexToElasticsearchTranslator translator = > >>>>>> - new ElasticsearchRules.RexToElasticsearchTranslator( > >>>>>> - (JavaTypeFactory) getCluster().getTypeFactory(), > >>> inFields); > >>>>>> + new ElasticsearchRules.RexToElasticsearchTranslator( > >>>>>> + (JavaTypeFactory) > getCluster().getTypeFactory(), > >>>>> inFields); > >>>>>> > >>>>>> - final List<String> findItems = new ArrayList<>(); > >>>>>> - final List<String> scriptFieldItems = new ArrayList<>(); > >>>>>> + final List<String> fields = new ArrayList<>(); > >>>>>> + final List<String> scriptFields = new ArrayList<>(); > >>>>>> for (Pair<RexNode, String> pair: getNamedProjects()) { > >>>>>> final String name = pair.right; > >>>>>> final String expr = pair.left.accept(translator); > >>>>>> > >>>>>> if (expr.equals("\"" + name + "\"")) { > >>>>>> - findItems.add(ElasticsearchRules.quote(name)); > >>>>>> + fields.add(name); > >>>>>> } else if (expr.matches("\"literal\":.+")) { > >>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name) > >>>>>> - + ":{\"script\": " > >>>>>> - + expr.split(":")[1] + "}"); > >>>>>> + scriptFields.add(ElasticsearchRules.quote(name) > >>>>>> + + ":{\"script\": " > >>>>>> + + expr.split(":")[1] + "}"); > >>>>>> } else { > >>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name) > >>>>>> - + ":{\"script\":\"params._source." > >>>>>> - + expr.replaceAll("\"", "") + "\"}"); > >>>>>> + scriptFields.add(ElasticsearchRules.quote(name) > >>>>>> + + ":{\"script\":" > >>>>>> + // _source (ES2) vs params._source (ES5) > >>>>>> + + "\"" + > >>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "." > >>>>>> + + expr.replaceAll("\"", "") + "\"}"); > >>>>>> + } > >>>>>> + } > >>>>>> + > >>>>>> + StringBuilder query = new StringBuilder(); > >>>>>> + if (scriptFields.isEmpty()) { > >>>>>> + List<String> newList = Lists.transform(fields, new > >>>>> Function<String, String>() { > >>>>>> + @Nullable > >>>>>> + @Override public String apply(@Nullable String input) { > >>>>>> + return ElasticsearchRules.quote(input); > >>>>>> + } > >>>>>> + }); > >>>>>> + > >>>>>> + final String findString = String.join(", ", newList); > >>>>>> + query.append("\"_source\" : > [").append(findString).append("]"); > >>>>>> + } else { > >>>>>> + // if scripted fields are present, ES ignores _source > attribute > >>>>>> + for (String field: fields) { > >>>>>> + scriptFields.add(ElasticsearchRules.quote(field) + > >>>>> ":{\"script\": " > >>>>>> + // _source (ES2) vs params._source (ES5) > >>>>>> + + "\"" + > >>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "." > >>>>>> + + field + "\"}"); > >>>>>> } > >>>>>> + query.append("\"script_fields\": {" + String.join(", ", > >>>>> scriptFields) + "}"); > >>>>>> } > >>>>>> - final String findString = Util.toString(findItems, "", ", ", > ""); > >>>>>> - final String scriptFieldString = "\"script_fields\": {" > >>>>>> - + Util.toString(scriptFieldItems, "", ", ", "") + "}"; > >>>>>> - final String fieldString = "\"_source\" : [" + findString + "]" > >>>>>> - + ", " + scriptFieldString; > >>>>>> > >>>>>> for (String opfield : implementor.list) { > >>>>>> if (opfield.startsWith("\"_source\"")) { > >>>>>> implementor.list.remove(opfield); > >>>>>> } > >>>>>> } > >>>>>> - implementor.add(fieldString); > >>>>>> + implementor.add(query.toString()); > >>>>>> } > >>>>>> } > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml > >>>>>> index f24622c..6fbee03 100644 > >>>>>> --- a/elasticsearch2/pom.xml > >>>>>> +++ b/elasticsearch2/pom.xml > >>>>>> @@ -73,6 +73,12 @@ limitations under the License. > >>>>>> <version>${elasticsearch-java-driver.version}</version> > >>>>>> </dependency> > >>>>>> <dependency> > >>>>>> + <!-- Lang groovy dependency is needed for testing with > embedded > >>>>> ES (scripted fields like loc[0]) --> > >>>>>> + <groupId>org.elasticsearch.module</groupId> > >>>>>> + <artifactId>lang-groovy</artifactId> > >>>>>> + <scope>test</scope> > >>>>>> + </dependency> > >>>>>> + <dependency> > >>>>>> <groupId>com.carrotsearch</groupId> > >>>>>> <artifactId>hppc</artifactId> > >>>>>> <version>${hppc.version}</version> > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>> > >>> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>> index 84370ab..c3d2ac0 100644 > >>>>>> --- > >>>>> > >>> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>> +++ > >>>>> > >>> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit; > >>>>>> import java.util.Date; > >>>>>> import java.util.Iterator; > >>>>>> import java.util.List; > >>>>>> +import java.util.Locale; > >>>>>> import java.util.Map; > >>>>>> > >>>>>> /** > >>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator > implements > >>>>> Enumerator<Object> { > >>>>>> private static Function1<SearchHit, Object[]> listGetter( > >>>>>> final List<Map.Entry<String, Class>> fields) { > >>>>>> return new Function1<SearchHit, Object[]>() { > >>>>>> - public Object[] apply(SearchHit searchHitFields) { > >>>>>> + public Object[] apply(SearchHit hit) { > >>>>>> Object[] objects = new Object[fields.size()]; > >>>>>> for (int i = 0; i < fields.size(); i++) { > >>>>>> final Map.Entry<String, Class> field = fields.get(i); > >>>>>> final String name = field.getKey(); > >>>>>> - if (searchHitFields.fields().isEmpty()) { > >>>>>> - objects[i] = > >>> convert(searchHitFields.getSource().get(name), > >>>>> field.getValue()); > >>>>>> + if (hit.fields().isEmpty()) { > >>>>>> + objects[i] = convert(hit.getSource().get(name), > >>>>> field.getValue()); > >>>>>> + } else if (hit.fields().containsKey(name)) { > >>>>>> + objects[i] = convert(hit.field(name).getValue(), > >>>>> field.getValue()); > >>>>>> } else { > >>>>>> - objects[i] = > >>>>> convert(searchHitFields.field(name).getValue(), field.getValue()); > >>>>>> + throw new IllegalStateException( > >>>>>> + String.format(Locale.getDefault(), "No result > for > >>>>> %s", field)); > >>>>>> } > >>>>>> } > >>>>>> return objects; > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>> > >>> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>> index 668402b..46e3fc5 100644 > >>>>>> --- > >>>>> > >>> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>> +++ > >>>>> > >>> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>>>>> @@ -22,6 +22,9 @@ import > org.apache.calcite.schema.impl.AbstractSchema; > >>>>>> > >>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor; > >>>>>> > >>>>>> +import com.google.common.annotations.VisibleForTesting; > >>>>>> +import com.google.common.base.Preconditions; > >>>>>> + > >>>>>> import com.google.common.collect.ImmutableList; > >>>>>> import com.google.common.collect.ImmutableMap; > >>>>>> > >>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends > >>>>> AbstractSchema > >>>>>> } > >>>>>> } > >>>>>> > >>>>>> + /** > >>>>>> + * Allows schema to be instantiated from existing elastic search > >>>>> client. > >>>>>> + * This constructor is used in tests. > >>>>>> + */ > >>>>>> + @VisibleForTesting > >>>>>> + Elasticsearch2Schema(Client client, String index) { > >>>>>> + this.client = Preconditions.checkNotNull(client, "client"); > >>>>>> + this.index = Preconditions.checkNotNull(index, "index"); > >>>>>> + } > >>>>>> + > >>>>>> @Override protected Map<String, Table> getTableMap() { > >>>>>> final ImmutableMap.Builder<String, Table> builder = > >>>>> ImmutableMap.builder(); > >>>>>> > >>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends > >>>>> AbstractSchema > >>>>>> > >>>>>> final List<DiscoveryNode> nodes = > >>>>> ImmutableList.copyOf(transportClient.connectedNodes()); > >>>>>> if (nodes.isEmpty()) { > >>>>>> - throw new RuntimeException("Cannot connect to any > elasticsearch > >>>>> nodes"); > >>>>>> + throw new IllegalStateException("Cannot connect to any > >>>>> elasticsearch node: " > >>>>>> + + transportNodes); > >>>>>> } > >>>>>> > >>>>>> client = transportClient; > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>> > >>> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>> index 636aa5f..2928835 100644 > >>>>>> --- > >>>>> > >>> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>> +++ > >>>>> > >>> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends > >>>>> AbstractElasticsearchTable { > >>>>>> this.client = client; > >>>>>> } > >>>>>> > >>>>>> + /** > >>>>>> + * ES version 2.x. To access document attributes ES2 uses {@code > >>>>> _source.foo} syntax. > >>>>>> + */ > >>>>>> + @Override protected String scriptedFieldPrefix() { > >>>>>> + return "_source"; > >>>>>> + } > >>>>>> + > >>>>>> @Override protected Enumerable<Object> find(String index, > List<String> > >>>>> ops, > >>>>>> - List<Map.Entry<String, Class>> fields) { > >>>>>> + > List<Map.Entry<String, > >>>>> Class>> fields) { > >>>>>> final String dbName = index; > >>>>>> > >>>>>> final String queryString = "{" + Util.toString(ops, "", ", ", "") + > >>>>> "}"; > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>> > >>> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>> new file mode 100644 > >>>>>> index 0000000..287e094 > >>>>>> --- /dev/null > >>>>>> +++ > >>>>> > >>> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>>>>> @@ -0,0 +1,395 @@ > >>>>>> +/* > >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or > more > >>>>>> + * contributor license agreements. See the NOTICE file distributed > >>> with > >>>>>> + * this work for additional information regarding copyright > ownership. > >>>>>> + * The ASF licenses this file to you under the Apache License, > Version > >>>>> 2.0 > >>>>>> + * (the "License"); you may not use this file except in compliance > >>> with > >>>>>> + * the License. You may obtain a copy of the License at > >>>>>> + * > >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>>>>> + * > >>>>>> + * Unless required by applicable law or agreed to in writing, > software > >>>>>> + * distributed under the License is distributed on an "AS IS" > BASIS, > >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >>>>> implied. > >>>>>> + * See the License for the specific language governing permissions > and > >>>>>> + * limitations under the License. > >>>>>> + */ > >>>>>> +package org.apache.calcite.adapter.elasticsearch2; > >>>>>> + > >>>>>> +import org.apache.calcite.jdbc.CalciteConnection; > >>>>>> +import org.apache.calcite.schema.SchemaPlus; > >>>>>> +import org.apache.calcite.schema.impl.ViewTable; > >>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro; > >>>>>> +import org.apache.calcite.test.CalciteAssert; > >>>>>> +import org.apache.calcite.test.ElasticChecker; > >>>>>> + > >>>>>> +import com.google.common.io.LineProcessor; > >>>>>> +import com.google.common.io.Resources; > >>>>>> + > >>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse; > >>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder; > >>>>>> +import org.elasticsearch.action.bulk.BulkResponse; > >>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder; > >>>>>> +import org.elasticsearch.common.xcontent.XContentFactory; > >>>>>> + > >>>>>> +import org.junit.BeforeClass; > >>>>>> +import org.junit.ClassRule; > >>>>>> +import org.junit.Test; > >>>>>> + > >>>>>> +import java.io.IOException; > >>>>>> +import java.nio.charset.StandardCharsets; > >>>>>> +import java.sql.Connection; > >>>>>> +import java.sql.DriverManager; > >>>>>> +import java.sql.SQLException; > >>>>>> +import java.util.Arrays; > >>>>>> +import java.util.Collections; > >>>>>> +import java.util.Locale; > >>>>>> + > >>>>>> +/** > >>>>>> + * Set of tests for ES adapter. Uses real instance via {@link > >>>>> EmbeddedElasticRule}. Document > >>>>>> + * source is local {@code zips-mini.json} file (located in the > >>>>> classpath). > >>>>>> + */ > >>>>>> +public class ElasticSearch2AdapterTest { > >>>>>> + > >>>>>> + @ClassRule //init once for all tests > >>>>>> + public static final EmbeddedElasticRule NODE = > >>>>> EmbeddedElasticRule.create(); > >>>>>> + > >>>>>> + private static final String ZIPS = "zips"; > >>>>>> + > >>>>>> + /** > >>>>>> + * Used to create {@code zips} index and insert some data > >>>>>> + */ > >>>>>> + @BeforeClass > >>>>>> + public static void setupInstance() throws Exception { > >>>>>> + // define mapping so fields are searchable (term query) > >>>>>> + XContentBuilder mapping = > >>>>> XContentFactory.jsonBuilder().startObject() > >>>>>> + .startObject("properties") > >>>>>> + .startObject("city").field("type", "string") > >>>>>> + .field("index", "not_analyzed").endObject() > >>>>>> + .startObject("state").field("type", "string") > >>>>>> + .field("index", "not_analyzed").endObject() > >>>>>> + .startObject("pop").field("type", "long").endObject() > >>>>>> + .endObject() > >>>>>> + .endObject(); > >>>>>> + > >>>>>> + // create index > >>>>>> + NODE.client().admin().indices() > >>>>>> + .prepareCreate(ZIPS) > >>>>>> + .addMapping(ZIPS, mapping) > >>>>>> + .get(); > >>>>>> + > >>>>>> + BulkRequestBuilder bulk = > >>>>> NODE.client().prepareBulk().setRefresh(true); > >>>>>> + > >>>>>> + // load records from file > >>>>>> + > >>>>> > >>> > Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"), > >>>>>> + StandardCharsets.UTF_8, new LineProcessor<Void>() { > >>>>>> + @Override public boolean processLine(String line) > throws > >>>>> IOException { > >>>>>> + line = line.replaceAll("_id", "id"); // _id is a > >>>>> reserved attribute in ES > >>>>>> + bulk.add(NODE.client().prepareIndex(ZIPS, > >>>>> ZIPS).setSource(line)); > >>>>>> + return true; > >>>>>> + } > >>>>>> + > >>>>>> + @Override public Void getResult() { > >>>>>> + return null; > >>>>>> + } > >>>>>> + }); > >>>>>> + > >>>>>> + if (bulk.numberOfActions() == 0) { > >>>>>> + throw new IllegalStateException("No records to be indexed"); > >>>>>> + } > >>>>>> + > >>>>>> + BulkResponse response = bulk.execute().get(); > >>>>>> + > >>>>>> + if (response.hasFailures()) { > >>>>>> + throw new IllegalStateException( > >>>>>> + String.format(Locale.getDefault(), "Failed to > populate > >>>>> %s:\n%s", NODE.httpAddress(), > >>>>>> + > >>>>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed) > >>>>>> + > >>>>> > >>> > .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>"))); > >>>>>> + } > >>>>>> + > >>>>>> + } > >>>>>> + > >>>>>> + private CalciteAssert.ConnectionFactory newConnectionFactory() { > >>>>>> + return new CalciteAssert.ConnectionFactory() { > >>>>>> + @Override public Connection createConnection() throws > >>>>> SQLException { > >>>>>> + final Connection connection = > >>>>> DriverManager.getConnection("jdbc:calcite:"); > >>>>>> + final SchemaPlus root = > >>>>> connection.unwrap(CalciteConnection.class).getRootSchema(); > >>>>>> + > >>>>>> + root.add("elastic", new Elasticsearch2Schema(NODE.client(), > >>>>> ZIPS)); > >>>>>> + > >>>>>> + // add calcite view programmatically > >>>>>> + final String viewSql = "select cast(_MAP['city'] AS > >>>>> varchar(20)) AS \"city\", " > >>>>>> + + " cast(_MAP['loc'][0] AS float) AS > \"longitude\",\n" > >>>>>> + + " cast(_MAP['loc'][1] AS float) AS > \"latitude\",\n" > >>>>>> + + " cast(_MAP['pop'] AS integer) AS \"pop\", " > >>>>>> + + " cast(_MAP['state'] AS varchar(2)) AS \"state\", > " > >>>>>> + + " cast(_MAP['id'] AS varchar(5)) AS \"id\" " > >>>>>> + + "from \"elastic\".\"zips\""; > >>>>>> + > >>>>>> + ViewTableMacro macro = ViewTable.viewMacro(root, viewSql, > >>>>>> + Collections.singletonList("elastic"), > >>>>> Arrays.asList("elastic", "view"), false); > >>>>>> + root.add("ZIPS", macro); > >>>>>> + > >>>>>> + return connection; > >>>>>> + } > >>>>>> + }; > >>>>>> + } > >>>>>> + > >>>>>> + private CalciteAssert.AssertThat calciteAssert() { > >>>>>> + return CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()); > >>>>>> + } > >>>>>> + > >>>>>> + /** > >>>>>> + * Tests using calcite view > >>>>>> + */ > >>>>>> + @Test > >>>>>> + public void view() throws Exception { > >>>>>> + calciteAssert() > >>>>>> + .query("select * from zips where \"city\" = 'BROOKLYN'") > >>>>>> + .returns("city=BROOKLYN; longitude=-73.956985; > >>>>> latitude=40.646694; " > >>>>>> + + "pop=111396; state=NY; id=11226\n") > >>>>>> + .returnsCount(1); > >>>>>> + } > >>>>>> + > >>>>>> + @Test > >>>>>> + public void emptyResult() { > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from zips limit 0") > >>>>>> + .returnsCount(0); > >>>>>> + > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from \"elastic\".\"zips\" where > >>>>> _MAP['Foo'] = '_MISSING_'") > >>>>>> + .returnsCount(0); > >>>>>> + } > >>>>>> + > >>>>>> + @Test > >>>>>> + public void basic() throws Exception { > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from \"elastic\".\"zips\" where > >>>>> _MAP['city'] = 'BROOKLYN'") > >>>>>> + .returnsCount(1); > >>>>>> + > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from \"elastic\".\"zips\" where" > >>>>>> + + " _MAP['city'] in ('BROOKLYN', > 'WASHINGTON')") > >>>>>> + .returnsCount(2); > >>>>>> + > >>>>>> + // lower-case > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from \"elastic\".\"zips\" where " > >>>>>> + + "_MAP['city'] in ('brooklyn', 'Brooklyn', > >>>>> 'BROOK') ") > >>>>>> + .returnsCount(0); > >>>>>> + > >>>>>> + // missing field > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from \"elastic\".\"zips\" where > >>>>> _MAP['CITY'] = 'BROOKLYN'") > >>>>>> + .returnsCount(0); > >>>>>> + > >>>>>> + // limit works > >>>>>> + CalciteAssert.that() > >>>>>> + .with(newConnectionFactory()) > >>>>>> + .query("select * from \"elastic\".\"zips\" limit 42") > >>>>>> + .returnsCount(42); > >>>>>> + > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testSort() { > >>>>>> + final String explain = > "PLAN=ElasticsearchToEnumerableConverter\n" > >>>>>> + + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n" > >>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0, > >>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), > >>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], > >>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, > >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) > >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE > \"ISO-8859-1$en_US$primary\"])\n" > >>>>>> + + " ElasticsearchTableScan(table=[[elastic, > zips]])"; > >>>>>> + > >>>>>> + calciteAssert() > >>>>>> + .query("select * from zips order by \"state\"") > >>>>>> + .returnsCount(10) > >>>>>> + .explainContains(explain); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testSortLimit() { > >>>>>> + final String sql = "select \"state\", \"id\" from zips\n" > >>>>>> + + "order by \"state\", \"id\" offset 2 rows fetch next > 3 > >>>>> rows only"; > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .returnsUnordered("state=AK; id=99801", > >>>>>> + "state=AL; id=35215", > >>>>>> + "state=AL; id=35401") > >>>>>> + .queryContains( > >>>>>> + ElasticChecker.elasticsearchChecker( > >>>>>> + "\"_source\" : [\"state\", \"id\"]", > >>>>>> + "\"sort\": [ {\"state\": \"asc\"}, > >>> {\"id\": > >>>>> \"asc\"}]", > >>>>>> + "\"from\": 2", > >>>>>> + "\"size\": 3")); > >>>>>> + } > >>>>>> + > >>>>>> + > >>>>>> + > >>>>>> + @Test public void testOffsetLimit() { > >>>>>> + final String sql = "select \"state\", \"id\" from zips\n" > >>>>>> + + "offset 2 fetch next 3 rows only"; > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .runs() > >>>>>> + .queryContains( > >>>>>> + ElasticChecker.elasticsearchChecker( > >>>>>> + "\"from\": 2", > >>>>>> + "\"size\": 3", > >>>>>> + "\"_source\" : [\"state\", \"id\"]")); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testLimit() { > >>>>>> + final String sql = "select \"state\", \"id\" from zips\n" > >>>>>> + + "fetch next 3 rows only"; > >>>>>> + > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .runs() > >>>>>> + .queryContains( > >>>>>> + ElasticChecker.elasticsearchChecker( > >>>>>> + "\"size\": 3", > >>>>>> + "\"_source\" : [\"state\", \"id\"]")); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testFilterSort() { > >>>>>> + final String sql = "select * from zips\n" > >>>>>> + + "where \"state\" = 'CA' and \"id\" >= '70000'\n" > >>>>>> + + "order by \"state\", \"id\""; > >>>>>> + final String explain = > "PLAN=ElasticsearchToEnumerableConverter\n" > >>>>>> + + " ElasticsearchSort(sort0=[$4], sort1=[$5], > dir0=[ASC], > >>>>> dir1=[ASC])\n" > >>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0, > >>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), > >>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], > >>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, > >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) > >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE > \"ISO-8859-1$en_US$primary\"])\n" > >>>>>> + + " > >>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0, > >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0, > 'id')):VARCHAR(5) > >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\", > >>>>> '70000'))])\n" > >>>>>> + + " ElasticsearchTableScan(table=[[elastic, > >>> zips]])"; > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .returnsOrdered("city=LOS ANGELES; > longitude=-118.258189; > >>>>> latitude=34.007856; " > >>>>>> + + "pop=96074; state=CA; id=90011", > >>>>>> + "city=BELL GARDENS; longitude=-118.17205; > >>>>> latitude=33.969177; " > >>>>>> + + "pop=99568; state=CA; id=90201", > >>>>>> + "city=NORWALK; longitude=-118.081767; > >>>>> latitude=33.90564; " > >>>>>> + + "pop=94188; state=CA; id=90650") > >>>>>> + .queryContains( > >>>>>> + ElasticChecker.elasticsearchChecker("\"query\" > : " > >>>>>> + + > >>>>> "{\"constant_score\":{\"filter\":{\"bool\":" > >>>>>> + + > >>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}}," > >>>>>> + + > >>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}", > >>>>>> + "\"script_fields\": > >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>>>>> + + > >>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " > >>>>>> + + "\"city\":{\"script\": > >>>>> \"_source.city\"}, " > >>>>>> + + "\"pop\":{\"script\": > >>>>> \"_source.pop\"}, " > >>>>>> + + "\"state\":{\"script\": > >>>>> \"_source.state\"}, " > >>>>>> + + "\"id\":{\"script\": \"_ > >>> source.id > >>>>> \"}}", > >>>>>> + "\"sort\": [ {\"state\": \"asc\"}, > >>> {\"id\": > >>>>> \"asc\"}]")) > >>>>>> + .explainContains(explain); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testFilterSortDesc() { > >>>>>> + final String sql = "select * from zips\n" > >>>>>> + + "where \"pop\" BETWEEN 95000 AND 100000\n" > >>>>>> + + "order by \"state\" desc, \"pop\""; > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .limit(4) > >>>>>> + .returnsOrdered( > >>>>>> + "city=LOS ANGELES; longitude=-118.258189; > >>>>> latitude=34.007856; pop=96074; state=CA; id=90011", > >>>>>> + "city=BELL GARDENS; longitude=-118.17205; > >>>>> latitude=33.969177; pop=99568; state=CA; id=90201"); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testFilterRedundant() { > >>>>>> + final String sql = "select * from zips\n" > >>>>>> + + "where \"state\" > 'CA' and \"state\" < 'AZ' and > >>>>> \"state\" = 'OK'"; > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .runs() > >>>>>> + .queryContains( > >>>>>> + ElasticChecker.elasticsearchChecker("" > >>>>>> + + "\"query\" : > >>>>> {\"constant_score\":{\"filter\":{\"bool\":" > >>>>>> + + > >>>>> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}", > >>>>>> + "\"script_fields\": > >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>>>>> + + > >>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " > >>>>>> + + "\"city\":{\"script\": > >>>>> \"_source.city\"}, " > >>>>>> + + "\"pop\":{\"script\": > >>>>> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, " > >>>>>> + + > "\"id\":{\"script\": > >>>>> \"_source.id\"}}" > >>>>>> + )); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testInPlan() { > >>>>>> + final String[] searches = { > >>>>>> + "\"query\" : > >>>>> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":" > >>>>>> + + > >>>>> > >>> > "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":" > >>>>>> + + "{\"pop\":99568}}]}}]}}}}", > >>>>>> + "\"script_fields\": > >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>>>>> + + "\"latitude\":{\"script\":\"_source.loc[1]\"}, > " > >>>>>> + + "\"city\":{\"script\": \"_source.city\"}, " > >>>>>> + + "\"pop\":{\"script\": \"_source.pop\"}, " > >>>>>> + + "\"state\":{\"script\": \"_source.state\"}, " > >>>>>> + + "\"id\":{\"script\": \"_source.id\"}}" > >>>>>> + }; > >>>>>> + > >>>>>> + calciteAssert() > >>>>>> + .query("select * from zips where \"pop\" in (96074, > >>> 99568)") > >>>>>> + .returnsUnordered( > >>>>>> + "city=BELL GARDENS; longitude=-118.17205; > >>>>> latitude=33.969177; pop=99568; state=CA; id=90201", > >>>>>> + "city=LOS ANGELES; longitude=-118.258189; > >>>>> latitude=34.007856; pop=96074; state=CA; id=90011" > >>>>>> + ) > >>>>>> + > >>>>> .queryContains(ElasticChecker.elasticsearchChecker(searches)); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testZips() { > >>>>>> + calciteAssert() > >>>>>> + .query("select \"state\", \"city\" from zips") > >>>>>> + .returnsCount(10); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testProject() { > >>>>>> + final String sql = "select \"state\", \"city\", 0 as > \"zero\"\n" > >>>>>> + + "from zips\n" > >>>>>> + + "order by \"state\", \"city\""; > >>>>>> + > >>>>>> + calciteAssert() > >>>>>> + .query(sql) > >>>>>> + .limit(2) > >>>>>> + .returnsUnordered("state=AK; city=ANCHORAGE; zero=0", > >>>>>> + "state=AK; city=FAIRBANKS; zero=0") > >>>>>> + .queryContains( > >>>>>> + > >>>>> ElasticChecker.elasticsearchChecker("\"script_fields\": " > >>>>>> + + "{\"zero\":{\"script\": > \"0\"}, > >>> " > >>>>>> + + "\"state\":{\"script\": > >>>>> \"_source.state\"}, " > >>>>>> + + "\"city\":{\"script\": > >>>>> \"_source.city\"}}", > >>>>>> + "\"sort\": [ {\"state\": \"asc\"}, > >>>>> {\"city\": \"asc\"}]")); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testFilter() { > >>>>>> + final String explain = > "PLAN=ElasticsearchToEnumerableConverter\n" > >>>>>> + + " ElasticsearchProject(state=[CAST(ITEM($0, > >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\"], city=[CAST(ITEM($0, > 'city')):VARCHAR(20) > >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE > \"ISO-8859-1$en_US$primary\"])\n" > >>>>>> + + " ElasticsearchFilter(condition=[=(CAST(ITEM($0, > >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >>>>> \"ISO-8859-1$en_US$primary\", 'CA')])\n" > >>>>>> + + " ElasticsearchTableScan(table=[[elastic, > zips]])"; > >>>>>> + calciteAssert() > >>>>>> + .query("select \"state\", \"city\" from zips where > >>>>> \"state\" = 'CA'") > >>>>>> + .limit(3) > >>>>>> + .returnsUnordered("state=CA; city=BELL GARDENS", > >>>>>> + "state=CA; city=LOS ANGELES", > >>>>>> + "state=CA; city=NORWALK") > >>>>>> + .explainContains(explain); > >>>>>> + } > >>>>>> + > >>>>>> + @Test public void testFilterReversed() { > >>>>>> + calciteAssert() > >>>>>> + .query("select \"state\", \"city\" from zips where 'WI' < > >>>>> \"state\" order by \"city\"") > >>>>>> + .limit(2) > >>>>>> + .returnsUnordered("state=WV; city=BECKLEY", > >>>>>> + "state=WY; city=CHEYENNE"); > >>>>>> + calciteAssert() > >>>>>> + .query("select \"state\", \"city\" from zips where > \"state\" > >>>>>> 'WI' order by \"city\"") > >>>>>> + .limit(2) > >>>>>> + .returnsUnordered("state=WV; city=BECKLEY", > >>>>>> + "state=WY; city=CHEYENNE"); > >>>>>> + } > >>>>>> + > >>>>>> +} > >>>>>> + > >>>>>> +// End ElasticSearch2AdapterTest.java > >>>>>> > >>>>>> > >>>>> > >>> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>>>>> > ---------------------------------------------------------------------- > >>>>>> diff --git > >>>>> > >>> > a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>>>> > >>> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>>>>> new file mode 100644 > >>>>>> index 0000000..4474add > >>>>>> --- /dev/null > >>>>>> +++ > >>>>> > >>> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>>>>> @@ -0,0 +1,147 @@ > >>>>>> +/* > >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or > more > >>>>>> + * contributor license agreements. See the NOTICE file distributed > >>> with > >>>>>> + * this work for additional information regarding copyright > ownership. > >>>>>> + * The ASF licenses this file to you under the Apache License, > Version > >>>>> 2.0 > >>>>>> + * (the "License"); you may not use this file except in compliance > >>> with > >>>>>> + * the License. You may obtain a copy of the License at > >>>>>> + * > >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>>>>> + * > >>>>>> + * Unless required by applicable law or agreed to in writing, > software > >>>>>> + * distributed under the License is distributed on an "AS IS" > BASIS, > >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >>>>> implied. > >>>>>> + * See the License for the specific language governing permissions > and > >>>>>> + * limitations under the License. > >>>>>> + */ > >>>>>> +package org.apache.calcite.adapter.elasticsearch2; > >>>>>> + > >>>>>> +import com.google.common.base.Preconditions; > >>>>>> +import com.google.common.io.Files; > >>>>>> + > >>>>>> +import org.elasticsearch.Version; > >>>>>> +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo; > >>>>>> +import org.elasticsearch.action.admin.cluster.node.info > >>>>> .NodesInfoResponse; > >>>>>> +import org.elasticsearch.client.Client; > >>>>>> +import org.elasticsearch.common.settings.Settings; > >>>>>> +import org.elasticsearch.common.transport.TransportAddress; > >>>>>> +import org.elasticsearch.node.Node; > >>>>>> +import org.elasticsearch.node.internal.InternalSettingsPreparer; > >>>>>> +import org.elasticsearch.plugins.Plugin; > >>>>>> +import org.elasticsearch.script.groovy.GroovyPlugin; > >>>>>> + > >>>>>> +import java.io.File; > >>>>>> +import java.util.Arrays; > >>>>>> +import java.util.Collection; > >>>>>> +import java.util.Collections; > >>>>>> + > >>>>>> +/** > >>>>>> + * Represents a single elastic search node which can run embedded > in a > >>>>> java application. > >>>>>> + * Intended for unit and integration tests. Settings and plugins > are > >>>>> crafted for Calcite. > >>>>>> + */ > >>>>>> +class EmbeddedElasticNode implements AutoCloseable { > >>>>>> + > >>>>>> + private final LocalNode node; > >>>>>> + private volatile boolean isStarted; > >>>>>> + > >>>>>> + private EmbeddedElasticNode(LocalNode node) { > >>>>>> + this.node = Preconditions.checkNotNull(node, "node"); > >>>>>> + } > >>>>>> + > >>>>>> + /** > >>>>>> + * Having separate class to expose (protected) constructor which > >>>>> allows to install > >>>>>> + * different plugins. In our case it is {@code GroovyPlugin} for > >>>>> scripted fields like > >>>>>> + * {@code loc[0]} or {@code loc[1]['foo']}. > >>>>>> + */ > >>>>>> + private static class LocalNode extends Node { > >>>>>> + private LocalNode(Settings settings, Collection<Class<? extends > >>>>> Plugin>> classpathPlugins) { > >>>>>> + super(InternalSettingsPreparer.prepareEnvironment(settings, > >>> null), > >>>>>> + Version.CURRENT, > >>>>>> + classpathPlugins); > >>>>>> + } > >>>>>> + } > >>>>>> + > >>>>>> + /** > >>>>>> + * Creates an instance with existing settings > >>>>>> + */ > >>>>>> + private static EmbeddedElasticNode create(Settings settings) { > >>>>>> + // ensure GroovyPlugin is installed or otherwise scripted > fields > >>>>> would not work > >>>>>> + LocalNode node = new LocalNode(settings, > >>>>> Collections.singleton(GroovyPlugin.class)); > >>>>>> + return new EmbeddedElasticNode(node); > >>>>>> + } > >>>>>> + > >>>>>> + /** > >>>>>> + * Creates elastic node as single member of a cluster. Node will > not > >>>>> be started > >>>>>> + * unless {@link #start()} is explicitly called. > >>>>>> + */ > >>>>>> + public static EmbeddedElasticNode create() { > >>>>>> + File data = Files.createTempDir(); > >>>>>> + data.deleteOnExit(); > >>>>>> + File home = Files.createTempDir(); > >>>>>> + home.deleteOnExit(); > >>>>>> + > >>>>>> + Settings settings = Settings.builder() > >>>>>> + .put("node.name", "fake-elastic") > >>>>>> + .put("path.home", home.getAbsolutePath()) > >>>>>> + .put("path.data", data.getAbsolutePath()) > >>>>>> + .put("script.inline", true) // requires GroovyPlugin > >>>>>> + .put("script.indexed", true) // requires GroovyPlugin > >>>>>> + > .put("cluster.routing.allocation.disk.threshold_enabled", > >>>>> false) > >>>>>> + .put("node.local", true) > >>>>>> + .put("node.data", true) > >>>>>> + .put("network.host", "localhost") > >>>>>> + .build(); > >>>>>> + > >>>>>> + return create(settings); > >>>>>> + } > >>>>>> + > >>>>>> + /** > >>>>>> + * Starts current node > >>>>>> + */ > >>>>>> + public void start() { > >>>>>> + Preconditions.checkState(!isStarted, "already started"); > >>>>>> + node.start(); > >>>
