Same failure for me on JDK 8/macOS. The most likely other variable is maven: I have 3.5.2 on macOS, 3.5.3 on ubuntu.
I’m mystified why no one else is seeing javadoc failures on macOS. Julian > On Jun 22, 2018, at 4:35 PM, Julian Hyde <[email protected]> wrote: > > On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following failure: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on project > calcite-mongodb: Error generating maven-javadoc-plugin:3.0.1:test-javadoc > report: > [ERROR] Exit code: 1 - > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22: > error: cannot find symbol > [ERROR] import org.apache.calcite.test.CalciteAssert; > [ERROR] ^ > [ERROR] symbol: class CalciteAssert > [ERROR] location: package org.apache.calcite.test > [ERROR] > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137: > error: package CalciteAssert does not exist > [ERROR] private CalciteAssert.AssertThat assertModel(String model) { > [ERROR] ^ > [ERROR] > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145: > error: package CalciteAssert does not exist > [ERROR] private CalciteAssert.AssertThat assertModel(URL url) { > [ERROR] ^ > [ERROR] > /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43: > error: reference not found > [ERROR] /** Similar to {@link CalciteAssert#checkResultUnordered}, but > filters strings > [ERROR] ^ > > > >> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <[email protected]> wrote: >> >> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is JDK 8 vs >> 9 ? >> >> # JDK8 (oracle) / macOS >> $ mvn clean -DskipTests site >> >> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <[email protected]> wrote: >> >>> I just tried it again. The following fails for me (jdk9, ubuntu): >>> >>> $ git checkout c12cb4b0de >>> $ mvn clean -DskipTests site >>> >>> Constructing Javadoc information... >>> 1 error >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] Reactor Summary: >>> [INFO] >>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE [02:29 >>> min] >>> [INFO] Calcite Linq4j ..................................... SKIPPED >>> [INFO] Calcite Core ....................................... SKIPPED >>> [INFO] Calcite Cassandra .................................. SKIPPED >>> [INFO] Calcite Druid ...................................... SKIPPED >>> [INFO] Calcite Elasticsearch .............................. SKIPPED >>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED >>> [INFO] Calcite Examples ................................... SKIPPED >>> [INFO] Calcite Example CSV ................................ SKIPPED >>> [INFO] Calcite Example Function ........................... SKIPPED >>> [INFO] Calcite File ....................................... SKIPPED >>> [INFO] Calcite Geode ...................................... SKIPPED >>> [INFO] Calcite MongoDB .................................... SKIPPED >>> [INFO] Calcite Pig ........................................ SKIPPED >>> [INFO] Calcite Piglet ..................................... SKIPPED >>> [INFO] Calcite Plus ....................................... SKIPPED >>> [INFO] Calcite Server ..................................... SKIPPED >>> [INFO] Calcite Spark ...................................... SKIPPED >>> [INFO] Calcite Splunk ..................................... SKIPPED >>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] BUILD FAILURE >>> [INFO] >>> ------------------------------------------------------------------------ >>> [INFO] Total time: 02:29 min >>> [INFO] Finished at: 2018-06-22T14:53:13-07:00 >>> [INFO] >>> ------------------------------------------------------------------------ >>> [ERROR] Failed to execute goal >>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on >>> project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate >>> report: >>> [ERROR] Exit code: 1 - >>> /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: >>> error: package org.elasticsearch.node.internal does not exist >>> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; >>> [ERROR] ^ >>> [ERROR] >>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options @packages >>> @argfile >>> [ERROR] >>> [ERROR] Refer to the generated Javadoc files in >>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir. >>> [ERROR] -> [Help 1] >>> [ERROR] >>> [ERROR] To see the full stack trace of the errors, re-run Maven with the >>> -e switch. >>> [ERROR] Re-run Maven using the -X switch to enable full debug logging. >>> [ERROR] >>> [ERROR] For more information about the errors and possible solutions, >>> please read the following articles: >>> [ERROR] [Help 1] >>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException >>> >>> >>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <[email protected]> wrote: >>>> >>>> Odd, mvn clean site still works fine for me. >>>> -- >>>> Michael Mior >>>> [email protected] >>>> >>>> >>>> >>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <[email protected]> a écrit : >>>> >>>>> Looks like this change broke “mvn site” (perhaps also “mvn >>>>> javadoc:test-javadoc”). >>>>> >>>>> [ERROR] Failed to execute goal >>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on >>>>> project calcite: Error generating >>> maven-javadoc-plugin:3.0.1:test-aggregate >>>>> report: >>>>> [ERROR] Exit code: 1 - >>>>> >>> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: >>>>> error: package org.elasticsearch.node.internal does not exist >>>>> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; >>>>> [ERROR] ^ >>>>> [ERROR] >>>>> >>>>>> On Jun 21, 2018, at 3:39 AM, [email protected] wrote: >>>>>> >>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of >>>>> ES adapter (Andrei Sereda) >>>>>> >>>>>> After discussion on dev-list Integration tests (for ES) have been >>>>> removed. They're now >>>>>> superseded by unit tests (which execute queries against a real elastic >>>>> instance) >>>>>> >>>>>> Added local file (zips-mini.json) which contains a small subset of >>>>> original zips.json >>>>>> (allows to bootstrap tests faster) >>>>>> >>>>>> Created separate ES JUnit rule which can be re-used across different >>>>> tests. >>>>>> >>>>>> Both v2 and v5 of ES adapters are supported. >>>>>> >>>>>> Close apache/calcite#716 >>>>>> >>>>>> >>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo >>>>>> Commit: http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0 >>>>>> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0 >>>>>> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0 >>>>>> >>>>>> Branch: refs/heads/master >>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d >>>>>> Parents: 37944bb >>>>>> Author: Andrei Sereda <[email protected]> >>>>>> Authored: Thu May 31 18:19:10 2018 -0400 >>>>>> Committer: Michael Mior <[email protected]> >>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400 >>>>>> >>>>>> ---------------------------------------------------------------------- >>>>>> .../AbstractElasticsearchTable.java | 12 + >>>>>> .../elasticsearch/ElasticsearchProject.java | 61 ++- >>>>>> elasticsearch2/pom.xml | 6 + >>>>>> .../Elasticsearch2Enumerator.java | 12 +- >>>>>> .../elasticsearch2/Elasticsearch2Schema.java | 16 +- >>>>>> .../elasticsearch2/Elasticsearch2Table.java | 9 +- >>>>>> .../ElasticSearch2AdapterTest.java | 395 >>> ++++++++++++++++++ >>>>>> .../elasticsearch2/EmbeddedElasticNode.java | 147 +++++++ >>>>>> .../elasticsearch2/EmbeddedElasticRule.java | 97 +++++ >>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ >>>>>> .../calcite/test/Elasticsearch2AdapterIT.java | 270 ------------- >>>>>> .../resources/elasticsearch-zips-model.json | 50 --- >>>>>> .../src/test/resources/zips-mini.json | 149 +++++++ >>>>>> elasticsearch5/pom.xml | 31 ++ >>>>>> .../elasticsearch5/Elasticsearch5Schema.java | 17 +- >>>>>> .../elasticsearch5/Elasticsearch5Table.java | 11 +- >>>>>> .../ElasticSearch5AdapterTest.java | 399 >>> +++++++++++++++++++ >>>>>> .../elasticsearch5/EmbeddedElasticNode.java | 153 +++++++ >>>>>> .../elasticsearch5/EmbeddedElasticRule.java | 98 +++++ >>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ >>>>>> .../calcite/test/Elasticsearch5AdapterIT.java | 270 ------------- >>>>>> .../resources/elasticsearch-zips-model.json | 50 --- >>>>>> elasticsearch5/src/test/resources/log4j2.xml | 16 + >>>>>> .../src/test/resources/zips-mini.json | 149 +++++++ >>>>>> pom.xml | 20 +- >>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-) >>>>>> ---------------------------------------------------------------------- >>>>>> >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>>>> >>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>>>>> index 0980469..8cc5933 100644 >>>>>> --- >>>>> >>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>>>>> +++ >>>>> >>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>>>>> @@ -75,6 +75,18 @@ public abstract class AbstractElasticsearchTable >>>>> extends AbstractQueryableTable >>>>>> relOptTable, this, null); >>>>>> } >>>>>> >>>>>> + /** >>>>>> + * In ES 5.x scripted fields start with {@code params._source.foo} >>>>> while in ES2.x >>>>>> + * {@code _source.foo}. Helper method to build correct query based >>> on >>>>> runtime version of elastic. >>>>>> + * >>>>>> + * @see <a href=" >>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source >>>>> variable</a> >>>>>> + * @see <a href=" >>>>> >>> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html >>> ">Scripted >>>>> Fields</a> >>>>>> + */ >>>>>> + protected String scriptedFieldPrefix() { >>>>>> + // this is default pattern starting 5.x >>>>>> + return "params._source"; >>>>>> + } >>>>>> + >>>>>> /** Executes a "find" operation on the underlying type. >>>>>> * >>>>>> * <p>For example, >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>>>> >>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>>>>> index b42abd7..961c8b0 100644 >>>>>> --- >>>>> >>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>>>>> +++ >>>>> >>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>>>>> @@ -27,11 +27,15 @@ import >>>>> org.apache.calcite.rel.metadata.RelMetadataQuery; >>>>>> import org.apache.calcite.rel.type.RelDataType; >>>>>> import org.apache.calcite.rex.RexNode; >>>>>> import org.apache.calcite.util.Pair; >>>>>> -import org.apache.calcite.util.Util; >>>>>> + >>>>>> +import com.google.common.base.Function; >>>>>> +import com.google.common.collect.Lists; >>>>>> >>>>>> import java.util.ArrayList; >>>>>> import java.util.List; >>>>>> >>>>>> +import javax.annotation.Nullable; >>>>>> + >>>>>> /** >>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project} >>>>>> * relational expression in Elasticsearch. >>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends Project >>>>> implements ElasticsearchRel { >>>>>> implementor.visitChild(0, getInput()); >>>>>> >>>>>> final List<String> inFields = >>>>>> - >>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); >>>>>> + >>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); >>>>>> final ElasticsearchRules.RexToElasticsearchTranslator translator = >>>>>> - new ElasticsearchRules.RexToElasticsearchTranslator( >>>>>> - (JavaTypeFactory) getCluster().getTypeFactory(), >>> inFields); >>>>>> + new ElasticsearchRules.RexToElasticsearchTranslator( >>>>>> + (JavaTypeFactory) getCluster().getTypeFactory(), >>>>> inFields); >>>>>> >>>>>> - final List<String> findItems = new ArrayList<>(); >>>>>> - final List<String> scriptFieldItems = new ArrayList<>(); >>>>>> + final List<String> fields = new ArrayList<>(); >>>>>> + final List<String> scriptFields = new ArrayList<>(); >>>>>> for (Pair<RexNode, String> pair: getNamedProjects()) { >>>>>> final String name = pair.right; >>>>>> final String expr = pair.left.accept(translator); >>>>>> >>>>>> if (expr.equals("\"" + name + "\"")) { >>>>>> - findItems.add(ElasticsearchRules.quote(name)); >>>>>> + fields.add(name); >>>>>> } else if (expr.matches("\"literal\":.+")) { >>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name) >>>>>> - + ":{\"script\": " >>>>>> - + expr.split(":")[1] + "}"); >>>>>> + scriptFields.add(ElasticsearchRules.quote(name) >>>>>> + + ":{\"script\": " >>>>>> + + expr.split(":")[1] + "}"); >>>>>> } else { >>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name) >>>>>> - + ":{\"script\":\"params._source." >>>>>> - + expr.replaceAll("\"", "") + "\"}"); >>>>>> + scriptFields.add(ElasticsearchRules.quote(name) >>>>>> + + ":{\"script\":" >>>>>> + // _source (ES2) vs params._source (ES5) >>>>>> + + "\"" + >>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "." >>>>>> + + expr.replaceAll("\"", "") + "\"}"); >>>>>> + } >>>>>> + } >>>>>> + >>>>>> + StringBuilder query = new StringBuilder(); >>>>>> + if (scriptFields.isEmpty()) { >>>>>> + List<String> newList = Lists.transform(fields, new >>>>> Function<String, String>() { >>>>>> + @Nullable >>>>>> + @Override public String apply(@Nullable String input) { >>>>>> + return ElasticsearchRules.quote(input); >>>>>> + } >>>>>> + }); >>>>>> + >>>>>> + final String findString = String.join(", ", newList); >>>>>> + query.append("\"_source\" : [").append(findString).append("]"); >>>>>> + } else { >>>>>> + // if scripted fields are present, ES ignores _source attribute >>>>>> + for (String field: fields) { >>>>>> + scriptFields.add(ElasticsearchRules.quote(field) + >>>>> ":{\"script\": " >>>>>> + // _source (ES2) vs params._source (ES5) >>>>>> + + "\"" + >>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "." >>>>>> + + field + "\"}"); >>>>>> } >>>>>> + query.append("\"script_fields\": {" + String.join(", ", >>>>> scriptFields) + "}"); >>>>>> } >>>>>> - final String findString = Util.toString(findItems, "", ", ", ""); >>>>>> - final String scriptFieldString = "\"script_fields\": {" >>>>>> - + Util.toString(scriptFieldItems, "", ", ", "") + "}"; >>>>>> - final String fieldString = "\"_source\" : [" + findString + "]" >>>>>> - + ", " + scriptFieldString; >>>>>> >>>>>> for (String opfield : implementor.list) { >>>>>> if (opfield.startsWith("\"_source\"")) { >>>>>> implementor.list.remove(opfield); >>>>>> } >>>>>> } >>>>>> - implementor.add(fieldString); >>>>>> + implementor.add(query.toString()); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml >>>>>> index f24622c..6fbee03 100644 >>>>>> --- a/elasticsearch2/pom.xml >>>>>> +++ b/elasticsearch2/pom.xml >>>>>> @@ -73,6 +73,12 @@ limitations under the License. >>>>>> <version>${elasticsearch-java-driver.version}</version> >>>>>> </dependency> >>>>>> <dependency> >>>>>> + <!-- Lang groovy dependency is needed for testing with embedded >>>>> ES (scripted fields like loc[0]) --> >>>>>> + <groupId>org.elasticsearch.module</groupId> >>>>>> + <artifactId>lang-groovy</artifactId> >>>>>> + <scope>test</scope> >>>>>> + </dependency> >>>>>> + <dependency> >>>>>> <groupId>com.carrotsearch</groupId> >>>>>> <artifactId>hppc</artifactId> >>>>>> <version>${hppc.version}</version> >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>>>> >>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>>>>> index 84370ab..c3d2ac0 100644 >>>>>> --- >>>>> >>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>>>>> +++ >>>>> >>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit; >>>>>> import java.util.Date; >>>>>> import java.util.Iterator; >>>>>> import java.util.List; >>>>>> +import java.util.Locale; >>>>>> import java.util.Map; >>>>>> >>>>>> /** >>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator implements >>>>> Enumerator<Object> { >>>>>> private static Function1<SearchHit, Object[]> listGetter( >>>>>> final List<Map.Entry<String, Class>> fields) { >>>>>> return new Function1<SearchHit, Object[]>() { >>>>>> - public Object[] apply(SearchHit searchHitFields) { >>>>>> + public Object[] apply(SearchHit hit) { >>>>>> Object[] objects = new Object[fields.size()]; >>>>>> for (int i = 0; i < fields.size(); i++) { >>>>>> final Map.Entry<String, Class> field = fields.get(i); >>>>>> final String name = field.getKey(); >>>>>> - if (searchHitFields.fields().isEmpty()) { >>>>>> - objects[i] = >>> convert(searchHitFields.getSource().get(name), >>>>> field.getValue()); >>>>>> + if (hit.fields().isEmpty()) { >>>>>> + objects[i] = convert(hit.getSource().get(name), >>>>> field.getValue()); >>>>>> + } else if (hit.fields().containsKey(name)) { >>>>>> + objects[i] = convert(hit.field(name).getValue(), >>>>> field.getValue()); >>>>>> } else { >>>>>> - objects[i] = >>>>> convert(searchHitFields.field(name).getValue(), field.getValue()); >>>>>> + throw new IllegalStateException( >>>>>> + String.format(Locale.getDefault(), "No result for >>>>> %s", field)); >>>>>> } >>>>>> } >>>>>> return objects; >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>>>> >>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>>>>> index 668402b..46e3fc5 100644 >>>>>> --- >>>>> >>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>>>>> +++ >>>>> >>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>>>>> @@ -22,6 +22,9 @@ import org.apache.calcite.schema.impl.AbstractSchema; >>>>>> >>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor; >>>>>> >>>>>> +import com.google.common.annotations.VisibleForTesting; >>>>>> +import com.google.common.base.Preconditions; >>>>>> + >>>>>> import com.google.common.collect.ImmutableList; >>>>>> import com.google.common.collect.ImmutableMap; >>>>>> >>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends >>>>> AbstractSchema >>>>>> } >>>>>> } >>>>>> >>>>>> + /** >>>>>> + * Allows schema to be instantiated from existing elastic search >>>>> client. >>>>>> + * This constructor is used in tests. >>>>>> + */ >>>>>> + @VisibleForTesting >>>>>> + Elasticsearch2Schema(Client client, String index) { >>>>>> + this.client = Preconditions.checkNotNull(client, "client"); >>>>>> + this.index = Preconditions.checkNotNull(index, "index"); >>>>>> + } >>>>>> + >>>>>> @Override protected Map<String, Table> getTableMap() { >>>>>> final ImmutableMap.Builder<String, Table> builder = >>>>> ImmutableMap.builder(); >>>>>> >>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends >>>>> AbstractSchema >>>>>> >>>>>> final List<DiscoveryNode> nodes = >>>>> ImmutableList.copyOf(transportClient.connectedNodes()); >>>>>> if (nodes.isEmpty()) { >>>>>> - throw new RuntimeException("Cannot connect to any elasticsearch >>>>> nodes"); >>>>>> + throw new IllegalStateException("Cannot connect to any >>>>> elasticsearch node: " >>>>>> + + transportNodes); >>>>>> } >>>>>> >>>>>> client = transportClient; >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>>>> >>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>>>>> index 636aa5f..2928835 100644 >>>>>> --- >>>>> >>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>>>>> +++ >>>>> >>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends >>>>> AbstractElasticsearchTable { >>>>>> this.client = client; >>>>>> } >>>>>> >>>>>> + /** >>>>>> + * ES version 2.x. To access document attributes ES2 uses {@code >>>>> _source.foo} syntax. >>>>>> + */ >>>>>> + @Override protected String scriptedFieldPrefix() { >>>>>> + return "_source"; >>>>>> + } >>>>>> + >>>>>> @Override protected Enumerable<Object> find(String index, List<String> >>>>> ops, >>>>>> - List<Map.Entry<String, Class>> fields) { >>>>>> + List<Map.Entry<String, >>>>> Class>> fields) { >>>>>> final String dbName = index; >>>>>> >>>>>> final String queryString = "{" + Util.toString(ops, "", ", ", "") + >>>>> "}"; >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>>>>> new file mode 100644 >>>>>> index 0000000..287e094 >>>>>> --- /dev/null >>>>>> +++ >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>>>>> @@ -0,0 +1,395 @@ >>>>>> +/* >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> + * contributor license agreements. See the NOTICE file distributed >>> with >>>>>> + * this work for additional information regarding copyright ownership. >>>>>> + * The ASF licenses this file to you under the Apache License, Version >>>>> 2.0 >>>>>> + * (the "License"); you may not use this file except in compliance >>> with >>>>>> + * the License. You may obtain a copy of the License at >>>>>> + * >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 >>>>>> + * >>>>>> + * Unless required by applicable law or agreed to in writing, software >>>>>> + * distributed under the License is distributed on an "AS IS" BASIS, >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>> implied. >>>>>> + * See the License for the specific language governing permissions and >>>>>> + * limitations under the License. >>>>>> + */ >>>>>> +package org.apache.calcite.adapter.elasticsearch2; >>>>>> + >>>>>> +import org.apache.calcite.jdbc.CalciteConnection; >>>>>> +import org.apache.calcite.schema.SchemaPlus; >>>>>> +import org.apache.calcite.schema.impl.ViewTable; >>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro; >>>>>> +import org.apache.calcite.test.CalciteAssert; >>>>>> +import org.apache.calcite.test.ElasticChecker; >>>>>> + >>>>>> +import com.google.common.io.LineProcessor; >>>>>> +import com.google.common.io.Resources; >>>>>> + >>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse; >>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder; >>>>>> +import org.elasticsearch.action.bulk.BulkResponse; >>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder; >>>>>> +import org.elasticsearch.common.xcontent.XContentFactory; >>>>>> + >>>>>> +import org.junit.BeforeClass; >>>>>> +import org.junit.ClassRule; >>>>>> +import org.junit.Test; >>>>>> + >>>>>> +import java.io.IOException; >>>>>> +import java.nio.charset.StandardCharsets; >>>>>> +import java.sql.Connection; >>>>>> +import java.sql.DriverManager; >>>>>> +import java.sql.SQLException; >>>>>> +import java.util.Arrays; >>>>>> +import java.util.Collections; >>>>>> +import java.util.Locale; >>>>>> + >>>>>> +/** >>>>>> + * Set of tests for ES adapter. Uses real instance via {@link >>>>> EmbeddedElasticRule}. Document >>>>>> + * source is local {@code zips-mini.json} file (located in the >>>>> classpath). >>>>>> + */ >>>>>> +public class ElasticSearch2AdapterTest { >>>>>> + >>>>>> + @ClassRule //init once for all tests >>>>>> + public static final EmbeddedElasticRule NODE = >>>>> EmbeddedElasticRule.create(); >>>>>> + >>>>>> + private static final String ZIPS = "zips"; >>>>>> + >>>>>> + /** >>>>>> + * Used to create {@code zips} index and insert some data >>>>>> + */ >>>>>> + @BeforeClass >>>>>> + public static void setupInstance() throws Exception { >>>>>> + // define mapping so fields are searchable (term query) >>>>>> + XContentBuilder mapping = >>>>> XContentFactory.jsonBuilder().startObject() >>>>>> + .startObject("properties") >>>>>> + .startObject("city").field("type", "string") >>>>>> + .field("index", "not_analyzed").endObject() >>>>>> + .startObject("state").field("type", "string") >>>>>> + .field("index", "not_analyzed").endObject() >>>>>> + .startObject("pop").field("type", "long").endObject() >>>>>> + .endObject() >>>>>> + .endObject(); >>>>>> + >>>>>> + // create index >>>>>> + NODE.client().admin().indices() >>>>>> + .prepareCreate(ZIPS) >>>>>> + .addMapping(ZIPS, mapping) >>>>>> + .get(); >>>>>> + >>>>>> + BulkRequestBuilder bulk = >>>>> NODE.client().prepareBulk().setRefresh(true); >>>>>> + >>>>>> + // load records from file >>>>>> + >>>>> >>> Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"), >>>>>> + StandardCharsets.UTF_8, new LineProcessor<Void>() { >>>>>> + @Override public boolean processLine(String line) throws >>>>> IOException { >>>>>> + line = line.replaceAll("_id", "id"); // _id is a >>>>> reserved attribute in ES >>>>>> + bulk.add(NODE.client().prepareIndex(ZIPS, >>>>> ZIPS).setSource(line)); >>>>>> + return true; >>>>>> + } >>>>>> + >>>>>> + @Override public Void getResult() { >>>>>> + return null; >>>>>> + } >>>>>> + }); >>>>>> + >>>>>> + if (bulk.numberOfActions() == 0) { >>>>>> + throw new IllegalStateException("No records to be indexed"); >>>>>> + } >>>>>> + >>>>>> + BulkResponse response = bulk.execute().get(); >>>>>> + >>>>>> + if (response.hasFailures()) { >>>>>> + throw new IllegalStateException( >>>>>> + String.format(Locale.getDefault(), "Failed to populate >>>>> %s:\n%s", NODE.httpAddress(), >>>>>> + >>>>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed) >>>>>> + >>>>> >>> .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>"))); >>>>>> + } >>>>>> + >>>>>> + } >>>>>> + >>>>>> + private CalciteAssert.ConnectionFactory newConnectionFactory() { >>>>>> + return new CalciteAssert.ConnectionFactory() { >>>>>> + @Override public Connection createConnection() throws >>>>> SQLException { >>>>>> + final Connection connection = >>>>> DriverManager.getConnection("jdbc:calcite:"); >>>>>> + final SchemaPlus root = >>>>> connection.unwrap(CalciteConnection.class).getRootSchema(); >>>>>> + >>>>>> + root.add("elastic", new Elasticsearch2Schema(NODE.client(), >>>>> ZIPS)); >>>>>> + >>>>>> + // add calcite view programmatically >>>>>> + final String viewSql = "select cast(_MAP['city'] AS >>>>> varchar(20)) AS \"city\", " >>>>>> + + " cast(_MAP['loc'][0] AS float) AS \"longitude\",\n" >>>>>> + + " cast(_MAP['loc'][1] AS float) AS \"latitude\",\n" >>>>>> + + " cast(_MAP['pop'] AS integer) AS \"pop\", " >>>>>> + + " cast(_MAP['state'] AS varchar(2)) AS \"state\", " >>>>>> + + " cast(_MAP['id'] AS varchar(5)) AS \"id\" " >>>>>> + + "from \"elastic\".\"zips\""; >>>>>> + >>>>>> + ViewTableMacro macro = ViewTable.viewMacro(root, viewSql, >>>>>> + Collections.singletonList("elastic"), >>>>> Arrays.asList("elastic", "view"), false); >>>>>> + root.add("ZIPS", macro); >>>>>> + >>>>>> + return connection; >>>>>> + } >>>>>> + }; >>>>>> + } >>>>>> + >>>>>> + private CalciteAssert.AssertThat calciteAssert() { >>>>>> + return CalciteAssert.that() >>>>>> + .with(newConnectionFactory()); >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Tests using calcite view >>>>>> + */ >>>>>> + @Test >>>>>> + public void view() throws Exception { >>>>>> + calciteAssert() >>>>>> + .query("select * from zips where \"city\" = 'BROOKLYN'") >>>>>> + .returns("city=BROOKLYN; longitude=-73.956985; >>>>> latitude=40.646694; " >>>>>> + + "pop=111396; state=NY; id=11226\n") >>>>>> + .returnsCount(1); >>>>>> + } >>>>>> + >>>>>> + @Test >>>>>> + public void emptyResult() { >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from zips limit 0") >>>>>> + .returnsCount(0); >>>>>> + >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from \"elastic\".\"zips\" where >>>>> _MAP['Foo'] = '_MISSING_'") >>>>>> + .returnsCount(0); >>>>>> + } >>>>>> + >>>>>> + @Test >>>>>> + public void basic() throws Exception { >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from \"elastic\".\"zips\" where >>>>> _MAP['city'] = 'BROOKLYN'") >>>>>> + .returnsCount(1); >>>>>> + >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from \"elastic\".\"zips\" where" >>>>>> + + " _MAP['city'] in ('BROOKLYN', 'WASHINGTON')") >>>>>> + .returnsCount(2); >>>>>> + >>>>>> + // lower-case >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from \"elastic\".\"zips\" where " >>>>>> + + "_MAP['city'] in ('brooklyn', 'Brooklyn', >>>>> 'BROOK') ") >>>>>> + .returnsCount(0); >>>>>> + >>>>>> + // missing field >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from \"elastic\".\"zips\" where >>>>> _MAP['CITY'] = 'BROOKLYN'") >>>>>> + .returnsCount(0); >>>>>> + >>>>>> + // limit works >>>>>> + CalciteAssert.that() >>>>>> + .with(newConnectionFactory()) >>>>>> + .query("select * from \"elastic\".\"zips\" limit 42") >>>>>> + .returnsCount(42); >>>>>> + >>>>>> + } >>>>>> + >>>>>> + @Test public void testSort() { >>>>>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>>>>> + + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n" >>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0, >>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), >>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], >>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>>>>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; >>>>>> + >>>>>> + calciteAssert() >>>>>> + .query("select * from zips order by \"state\"") >>>>>> + .returnsCount(10) >>>>>> + .explainContains(explain); >>>>>> + } >>>>>> + >>>>>> + @Test public void testSortLimit() { >>>>>> + final String sql = "select \"state\", \"id\" from zips\n" >>>>>> + + "order by \"state\", \"id\" offset 2 rows fetch next 3 >>>>> rows only"; >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .returnsUnordered("state=AK; id=99801", >>>>>> + "state=AL; id=35215", >>>>>> + "state=AL; id=35401") >>>>>> + .queryContains( >>>>>> + ElasticChecker.elasticsearchChecker( >>>>>> + "\"_source\" : [\"state\", \"id\"]", >>>>>> + "\"sort\": [ {\"state\": \"asc\"}, >>> {\"id\": >>>>> \"asc\"}]", >>>>>> + "\"from\": 2", >>>>>> + "\"size\": 3")); >>>>>> + } >>>>>> + >>>>>> + >>>>>> + >>>>>> + @Test public void testOffsetLimit() { >>>>>> + final String sql = "select \"state\", \"id\" from zips\n" >>>>>> + + "offset 2 fetch next 3 rows only"; >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .runs() >>>>>> + .queryContains( >>>>>> + ElasticChecker.elasticsearchChecker( >>>>>> + "\"from\": 2", >>>>>> + "\"size\": 3", >>>>>> + "\"_source\" : [\"state\", \"id\"]")); >>>>>> + } >>>>>> + >>>>>> + @Test public void testLimit() { >>>>>> + final String sql = "select \"state\", \"id\" from zips\n" >>>>>> + + "fetch next 3 rows only"; >>>>>> + >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .runs() >>>>>> + .queryContains( >>>>>> + ElasticChecker.elasticsearchChecker( >>>>>> + "\"size\": 3", >>>>>> + "\"_source\" : [\"state\", \"id\"]")); >>>>>> + } >>>>>> + >>>>>> + @Test public void testFilterSort() { >>>>>> + final String sql = "select * from zips\n" >>>>>> + + "where \"state\" = 'CA' and \"id\" >= '70000'\n" >>>>>> + + "order by \"state\", \"id\""; >>>>>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>>>>> + + " ElasticsearchSort(sort0=[$4], sort1=[$5], dir0=[ASC], >>>>> dir1=[ASC])\n" >>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0, >>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), >>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], >>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>>>>> + + " >>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0, >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0, 'id')):VARCHAR(5) >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\", >>>>> '70000'))])\n" >>>>>> + + " ElasticsearchTableScan(table=[[elastic, >>> zips]])"; >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .returnsOrdered("city=LOS ANGELES; longitude=-118.258189; >>>>> latitude=34.007856; " >>>>>> + + "pop=96074; state=CA; id=90011", >>>>>> + "city=BELL GARDENS; longitude=-118.17205; >>>>> latitude=33.969177; " >>>>>> + + "pop=99568; state=CA; id=90201", >>>>>> + "city=NORWALK; longitude=-118.081767; >>>>> latitude=33.90564; " >>>>>> + + "pop=94188; state=CA; id=90650") >>>>>> + .queryContains( >>>>>> + ElasticChecker.elasticsearchChecker("\"query\" : " >>>>>> + + >>>>> "{\"constant_score\":{\"filter\":{\"bool\":" >>>>>> + + >>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}}," >>>>>> + + >>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}", >>>>>> + "\"script_fields\": >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " >>>>>> + + >>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " >>>>>> + + "\"city\":{\"script\": >>>>> \"_source.city\"}, " >>>>>> + + "\"pop\":{\"script\": >>>>> \"_source.pop\"}, " >>>>>> + + "\"state\":{\"script\": >>>>> \"_source.state\"}, " >>>>>> + + "\"id\":{\"script\": \"_ >>> source.id >>>>> \"}}", >>>>>> + "\"sort\": [ {\"state\": \"asc\"}, >>> {\"id\": >>>>> \"asc\"}]")) >>>>>> + .explainContains(explain); >>>>>> + } >>>>>> + >>>>>> + @Test public void testFilterSortDesc() { >>>>>> + final String sql = "select * from zips\n" >>>>>> + + "where \"pop\" BETWEEN 95000 AND 100000\n" >>>>>> + + "order by \"state\" desc, \"pop\""; >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .limit(4) >>>>>> + .returnsOrdered( >>>>>> + "city=LOS ANGELES; longitude=-118.258189; >>>>> latitude=34.007856; pop=96074; state=CA; id=90011", >>>>>> + "city=BELL GARDENS; longitude=-118.17205; >>>>> latitude=33.969177; pop=99568; state=CA; id=90201"); >>>>>> + } >>>>>> + >>>>>> + @Test public void testFilterRedundant() { >>>>>> + final String sql = "select * from zips\n" >>>>>> + + "where \"state\" > 'CA' and \"state\" < 'AZ' and >>>>> \"state\" = 'OK'"; >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .runs() >>>>>> + .queryContains( >>>>>> + ElasticChecker.elasticsearchChecker("" >>>>>> + + "\"query\" : >>>>> {\"constant_score\":{\"filter\":{\"bool\":" >>>>>> + + >>>>> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}", >>>>>> + "\"script_fields\": >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " >>>>>> + + >>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " >>>>>> + + "\"city\":{\"script\": >>>>> \"_source.city\"}, " >>>>>> + + "\"pop\":{\"script\": >>>>> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, " >>>>>> + + "\"id\":{\"script\": >>>>> \"_source.id\"}}" >>>>>> + )); >>>>>> + } >>>>>> + >>>>>> + @Test public void testInPlan() { >>>>>> + final String[] searches = { >>>>>> + "\"query\" : >>>>> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":" >>>>>> + + >>>>> >>> "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":" >>>>>> + + "{\"pop\":99568}}]}}]}}}}", >>>>>> + "\"script_fields\": >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " >>>>>> + + "\"latitude\":{\"script\":\"_source.loc[1]\"}, " >>>>>> + + "\"city\":{\"script\": \"_source.city\"}, " >>>>>> + + "\"pop\":{\"script\": \"_source.pop\"}, " >>>>>> + + "\"state\":{\"script\": \"_source.state\"}, " >>>>>> + + "\"id\":{\"script\": \"_source.id\"}}" >>>>>> + }; >>>>>> + >>>>>> + calciteAssert() >>>>>> + .query("select * from zips where \"pop\" in (96074, >>> 99568)") >>>>>> + .returnsUnordered( >>>>>> + "city=BELL GARDENS; longitude=-118.17205; >>>>> latitude=33.969177; pop=99568; state=CA; id=90201", >>>>>> + "city=LOS ANGELES; longitude=-118.258189; >>>>> latitude=34.007856; pop=96074; state=CA; id=90011" >>>>>> + ) >>>>>> + >>>>> .queryContains(ElasticChecker.elasticsearchChecker(searches)); >>>>>> + } >>>>>> + >>>>>> + @Test public void testZips() { >>>>>> + calciteAssert() >>>>>> + .query("select \"state\", \"city\" from zips") >>>>>> + .returnsCount(10); >>>>>> + } >>>>>> + >>>>>> + @Test public void testProject() { >>>>>> + final String sql = "select \"state\", \"city\", 0 as \"zero\"\n" >>>>>> + + "from zips\n" >>>>>> + + "order by \"state\", \"city\""; >>>>>> + >>>>>> + calciteAssert() >>>>>> + .query(sql) >>>>>> + .limit(2) >>>>>> + .returnsUnordered("state=AK; city=ANCHORAGE; zero=0", >>>>>> + "state=AK; city=FAIRBANKS; zero=0") >>>>>> + .queryContains( >>>>>> + >>>>> ElasticChecker.elasticsearchChecker("\"script_fields\": " >>>>>> + + "{\"zero\":{\"script\": \"0\"}, >>> " >>>>>> + + "\"state\":{\"script\": >>>>> \"_source.state\"}, " >>>>>> + + "\"city\":{\"script\": >>>>> \"_source.city\"}}", >>>>>> + "\"sort\": [ {\"state\": \"asc\"}, >>>>> {\"city\": \"asc\"}]")); >>>>>> + } >>>>>> + >>>>>> + @Test public void testFilter() { >>>>>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>>>>> + + " ElasticsearchProject(state=[CAST(ITEM($0, >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\"], city=[CAST(ITEM($0, 'city')):VARCHAR(20) >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>>>>> + + " ElasticsearchFilter(condition=[=(CAST(ITEM($0, >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >>>>> \"ISO-8859-1$en_US$primary\", 'CA')])\n" >>>>>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; >>>>>> + calciteAssert() >>>>>> + .query("select \"state\", \"city\" from zips where >>>>> \"state\" = 'CA'") >>>>>> + .limit(3) >>>>>> + .returnsUnordered("state=CA; city=BELL GARDENS", >>>>>> + "state=CA; city=LOS ANGELES", >>>>>> + "state=CA; city=NORWALK") >>>>>> + .explainContains(explain); >>>>>> + } >>>>>> + >>>>>> + @Test public void testFilterReversed() { >>>>>> + calciteAssert() >>>>>> + .query("select \"state\", \"city\" from zips where 'WI' < >>>>> \"state\" order by \"city\"") >>>>>> + .limit(2) >>>>>> + .returnsUnordered("state=WV; city=BECKLEY", >>>>>> + "state=WY; city=CHEYENNE"); >>>>>> + calciteAssert() >>>>>> + .query("select \"state\", \"city\" from zips where \"state\" >>>>>> 'WI' order by \"city\"") >>>>>> + .limit(2) >>>>>> + .returnsUnordered("state=WV; city=BECKLEY", >>>>>> + "state=WY; city=CHEYENNE"); >>>>>> + } >>>>>> + >>>>>> +} >>>>>> + >>>>>> +// End ElasticSearch2AdapterTest.java >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>>>>> new file mode 100644 >>>>>> index 0000000..4474add >>>>>> --- /dev/null >>>>>> +++ >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>>>>> @@ -0,0 +1,147 @@ >>>>>> +/* >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> + * contributor license agreements. See the NOTICE file distributed >>> with >>>>>> + * this work for additional information regarding copyright ownership. >>>>>> + * The ASF licenses this file to you under the Apache License, Version >>>>> 2.0 >>>>>> + * (the "License"); you may not use this file except in compliance >>> with >>>>>> + * the License. You may obtain a copy of the License at >>>>>> + * >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 >>>>>> + * >>>>>> + * Unless required by applicable law or agreed to in writing, software >>>>>> + * distributed under the License is distributed on an "AS IS" BASIS, >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>> implied. >>>>>> + * See the License for the specific language governing permissions and >>>>>> + * limitations under the License. >>>>>> + */ >>>>>> +package org.apache.calcite.adapter.elasticsearch2; >>>>>> + >>>>>> +import com.google.common.base.Preconditions; >>>>>> +import com.google.common.io.Files; >>>>>> + >>>>>> +import org.elasticsearch.Version; >>>>>> +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo; >>>>>> +import org.elasticsearch.action.admin.cluster.node.info >>>>> .NodesInfoResponse; >>>>>> +import org.elasticsearch.client.Client; >>>>>> +import org.elasticsearch.common.settings.Settings; >>>>>> +import org.elasticsearch.common.transport.TransportAddress; >>>>>> +import org.elasticsearch.node.Node; >>>>>> +import org.elasticsearch.node.internal.InternalSettingsPreparer; >>>>>> +import org.elasticsearch.plugins.Plugin; >>>>>> +import org.elasticsearch.script.groovy.GroovyPlugin; >>>>>> + >>>>>> +import java.io.File; >>>>>> +import java.util.Arrays; >>>>>> +import java.util.Collection; >>>>>> +import java.util.Collections; >>>>>> + >>>>>> +/** >>>>>> + * Represents a single elastic search node which can run embedded in a >>>>> java application. >>>>>> + * Intended for unit and integration tests. Settings and plugins are >>>>> crafted for Calcite. >>>>>> + */ >>>>>> +class EmbeddedElasticNode implements AutoCloseable { >>>>>> + >>>>>> + private final LocalNode node; >>>>>> + private volatile boolean isStarted; >>>>>> + >>>>>> + private EmbeddedElasticNode(LocalNode node) { >>>>>> + this.node = Preconditions.checkNotNull(node, "node"); >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Having separate class to expose (protected) constructor which >>>>> allows to install >>>>>> + * different plugins. In our case it is {@code GroovyPlugin} for >>>>> scripted fields like >>>>>> + * {@code loc[0]} or {@code loc[1]['foo']}. >>>>>> + */ >>>>>> + private static class LocalNode extends Node { >>>>>> + private LocalNode(Settings settings, Collection<Class<? extends >>>>> Plugin>> classpathPlugins) { >>>>>> + super(InternalSettingsPreparer.prepareEnvironment(settings, >>> null), >>>>>> + Version.CURRENT, >>>>>> + classpathPlugins); >>>>>> + } >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Creates an instance with existing settings >>>>>> + */ >>>>>> + private static EmbeddedElasticNode create(Settings settings) { >>>>>> + // ensure GroovyPlugin is installed or otherwise scripted fields >>>>> would not work >>>>>> + LocalNode node = new LocalNode(settings, >>>>> Collections.singleton(GroovyPlugin.class)); >>>>>> + return new EmbeddedElasticNode(node); >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Creates elastic node as single member of a cluster. Node will not >>>>> be started >>>>>> + * unless {@link #start()} is explicitly called. >>>>>> + */ >>>>>> + public static EmbeddedElasticNode create() { >>>>>> + File data = Files.createTempDir(); >>>>>> + data.deleteOnExit(); >>>>>> + File home = Files.createTempDir(); >>>>>> + home.deleteOnExit(); >>>>>> + >>>>>> + Settings settings = Settings.builder() >>>>>> + .put("node.name", "fake-elastic") >>>>>> + .put("path.home", home.getAbsolutePath()) >>>>>> + .put("path.data", data.getAbsolutePath()) >>>>>> + .put("script.inline", true) // requires GroovyPlugin >>>>>> + .put("script.indexed", true) // requires GroovyPlugin >>>>>> + .put("cluster.routing.allocation.disk.threshold_enabled", >>>>> false) >>>>>> + .put("node.local", true) >>>>>> + .put("node.data", true) >>>>>> + .put("network.host", "localhost") >>>>>> + .build(); >>>>>> + >>>>>> + return create(settings); >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Starts current node >>>>>> + */ >>>>>> + public void start() { >>>>>> + Preconditions.checkState(!isStarted, "already started"); >>>>>> + node.start(); >>>>>> + this.isStarted = true; >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Returns current address to connect to with HTTP client. >>>>>> + */ >>>>>> + public TransportAddress httpAddress() { >>>>>> + Preconditions.checkState(isStarted, "node is not started"); >>>>>> + >>>>>> + NodesInfoResponse response = >>>>> client().admin().cluster().prepareNodesInfo() >>>>>> + .execute().actionGet(); >>>>>> + if (response.getNodes().length != 1) { >>>>>> + throw new IllegalStateException("Expected single node but got " >>>>>> + + response.getNodes().length); >>>>>> + } >>>>>> + NodeInfo node = response.getNodes()[0]; >>>>>> + return node.getHttp().address().boundAddresses()[0]; >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Exposes elastic >>>>>> + * <a href=" >>>>> >>> https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html >>> ">transport >>>>> client</a> >>>>>> + * >>>>>> + * (use of HTTP client is preferred). >>>>>> + */ >>>>>> + public Client client() { >>>>>> + Preconditions.checkState(isStarted, "node is not started"); >>>>>> + return node.client(); >>>>>> + } >>>>>> + >>>>>> + @Override public void close() throws Exception { >>>>>> + node.close(); >>>>>> + // cleanup data dirs >>>>>> + File data = new File(node.settings().get("path.data")); >>>>>> + File home = new File(node.settings().get("path.home")); >>>>>> + for (File file: Arrays.asList(data, home)) { >>>>>> + if (file.exists()) { >>>>>> + file.delete(); >>>>>> + } >>>>>> + } >>>>>> + } >>>>>> +} >>>>>> + >>>>>> +// End EmbeddedElasticNode.java >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>>>>> new file mode 100644 >>>>>> index 0000000..a633078 >>>>>> --- /dev/null >>>>>> +++ >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>>>>> @@ -0,0 +1,97 @@ >>>>>> +/* >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> + * contributor license agreements. See the NOTICE file distributed >>> with >>>>>> + * this work for additional information regarding copyright ownership. >>>>>> + * The ASF licenses this file to you under the Apache License, Version >>>>> 2.0 >>>>>> + * (the "License"); you may not use this file except in compliance >>> with >>>>>> + * the License. You may obtain a copy of the License at >>>>>> + * >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 >>>>>> + * >>>>>> + * Unless required by applicable law or agreed to in writing, software >>>>>> + * distributed under the License is distributed on an "AS IS" BASIS, >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>> implied. >>>>>> + * See the License for the specific language governing permissions and >>>>>> + * limitations under the License. >>>>>> + */ >>>>>> +package org.apache.calcite.adapter.elasticsearch2; >>>>>> + >>>>>> +import com.google.common.base.Preconditions; >>>>>> + >>>>>> +import org.elasticsearch.client.Client; >>>>>> +import org.elasticsearch.common.transport.TransportAddress; >>>>>> +import org.junit.rules.ExternalResource; >>>>>> + >>>>>> +/** >>>>>> + * Used to initialize a single elastic node. For performance reasons >>>>> (node startup costs), >>>>>> + * same instance is usually shared across multiple tests. >>>>>> + * >>>>>> + * This rule should be used as follows: >>>>>> + * <pre> >>>>>> + * {@code >>>>>> + * >>>>>> + * public class MyTest { >>>>>> + * @literal @ClassRule >>>>>> + * public static final ElasticSearchRule RULE = >>>>> ElasticSearchRule.create(); >>>>>> + * >>>>>> + * @literal @BeforeClass >>>>>> + * public void setup() { >>>>>> + * // ... populate instance >>>>>> + * } >>>>>> + * >>>>>> + * @literal @Test >>>>>> + * public void myTest() { >>>>>> + * TransportAddress address = RULE.httpAddress(); >>>>>> + * // .... >>>>>> + * } >>>>>> + * } >>>>>> + * } >>>>>> + * </pre> >>>>>> + * >>>>>> + * @see ExternalResource >>>>>> + */ >>>>>> +class EmbeddedElasticRule extends ExternalResource { >>>>>> + >>>>>> + private final EmbeddedElasticNode node; >>>>>> + >>>>>> + private EmbeddedElasticRule(EmbeddedElasticNode resource) { >>>>>> + this.node = Preconditions.checkNotNull(resource, "resource"); >>>>>> + } >>>>>> + >>>>>> + @Override protected void before() throws Throwable { >>>>>> + node.start(); >>>>>> + } >>>>>> + >>>>>> + @Override protected void after() { >>>>>> + try { >>>>>> + node.close(); >>>>>> + } catch (Exception e) { >>>>>> + throw new RuntimeException(e); >>>>>> + } >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Factory method to create this rule. >>>>>> + */ >>>>>> + public static EmbeddedElasticRule create() { >>>>>> + return new EmbeddedElasticRule(EmbeddedElasticNode.create()); >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * Exposes current ES transport client. >>>>>> + */ >>>>>> + Client client() { >>>>>> + return node.client(); >>>>>> + } >>>>>> + >>>>>> + /** >>>>>> + * HTTP address for rest clients (can be ES native or any other). >>>>>> + */ >>>>>> + TransportAddress httpAddress() { >>>>>> + return node.httpAddress(); >>>>>> + } >>>>>> + >>>>>> + >>>>>> +} >>>>>> + >>>>>> +// End EmbeddedElasticRule.java >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>>>>> new file mode 100644 >>>>>> index 0000000..21fc491 >>>>>> --- /dev/null >>>>>> +++ >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>>>>> @@ -0,0 +1,49 @@ >>>>>> +/* >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> + * contributor license agreements. See the NOTICE file distributed >>> with >>>>>> + * this work for additional information regarding copyright ownership. >>>>>> + * The ASF licenses this file to you under the Apache License, Version >>>>> 2.0 >>>>>> + * (the "License"); you may not use this file except in compliance >>> with >>>>>> + * the License. You may obtain a copy of the License at >>>>>> + * >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0 >>>>>> + * >>>>>> + * Unless required by applicable law or agreed to in writing, software >>>>>> + * distributed under the License is distributed on an "AS IS" BASIS, >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>> implied. >>>>>> + * See the License for the specific language governing permissions and >>>>>> + * limitations under the License. >>>>>> + */ >>>>>> +package org.apache.calcite.test; >>>>>> + >>>>>> +import com.google.common.base.Function; >>>>>> + >>>>>> +import java.util.List; >>>>>> + >>>>>> +import javax.annotation.Nullable; >>>>>> + >>>>>> +/** >>>>>> + * Internal util methods for ElasticSearch tests >>>>>> + */ >>>>>> +public class ElasticChecker { >>>>>> + >>>>>> + private ElasticChecker() {} >>>>>> + >>>>>> + >>>>>> + /** Returns a function that checks that a particular Elasticsearch >>>>> pipeline is >>>>>> + * generated to implement a query. */ >>>>>> + public static Function<List, Void> elasticsearchChecker(final >>>>> String... strings) { >>>>>> + return new Function<List, Void>() { >>>>>> + @Nullable >>>>>> + @Override public Void apply(@Nullable List actual) { >>>>>> + Object[] actualArray = actual == null || actual.isEmpty() ? >>> null >>>>>> + : ((List) actual.get(0)).toArray(); >>>>>> + CalciteAssert.assertArrayEqual("expected Elasticsearch query >>>>> not found", strings, >>>>>> + actualArray); >>>>>> + return null; >>>>>> + } >>>>>> + }; >>>>>> + } >>>>>> +} >>>>>> + >>>>>> +// End ElasticChecker.java >>>>>> >>>>>> >>>>> >>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>>>>> ---------------------------------------------------------------------- >>>>>> diff --git >>>>> >>> a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>>>> >>> b/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>>>>> deleted file mode 100644 >>>>>> index 4e0c2b6..0000000 >>>>>> --- >>>>> >>> a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>>>>> +++ /dev/null >>>>>> @@ -1,270 +0,0 @@ >>>>>> -/* >>>>>> - * Licensed to the Apache Software Foundation (ASF) under one or more >>>>>> - * contributor license agreements. See the NOTICE file distributed >>> with >>>>>> - * this work for additional information regarding copyright ownership. >>>>>> - * The ASF licenses this file to you under the Apache License, Version >>>>> 2.0 >>>>>> - * (the "License"); you may not use this file except in compliance >>> with >>>>>> - * the License. You may obtain a copy of the License at >>>>>> - * >>>>>> - * http://www.apache.org/licenses/LICENSE-2.0 >>>>>> - * >>>>>> - * Unless required by applicable law or agreed to in writing, software >>>>>> - * distributed under the License is distributed on an "AS IS" BASIS, >>>>>> - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>>> implied. >>>>>> - * See the License for the specific language governing permissions and >>>>>> - * limitations under the License. >>>>>> - */ >>>>>> -package org.apache.calcite.test; >>>>>> - >>>>>> -import org.apache.calcite.util.Util; >>>>>> - >>>>>> -import com.google.common.base.Function; >>>>>> -import com.google.common.collect.ImmutableMap; >>>>>> - >>>>>> -import org.junit.Test; >>>>>> - >>>>>> -import java.util.List; >>>>>> -import javax.annotation.Nullable; >>>>>> - >>>>>> -/** >>>>>> - * Tests for the {@code org.apache.calcite.adapter.elasticsearch2} >>>>> package. >>>>>> - * >>>>>> - * <p>Before calling this test, you need to populate Elasticsearch, as >>>>> follows: >>>>>> - * >>>>>> - * <blockquote><code> >>>>>> - * git clone https://github.com/vlsi/calcite-test-dataset<br> >>>>>> - * cd calcite-test-dataset<br> >>>>>> - * mvn install >>>>>> - * </code></blockquote> >>>>>> - * >>>>>> - * <p>This will create a virtual machine with Elastic >
