On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following failure:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on project
calcite-mongodb: Error generating maven-javadoc-plugin:3.0.1:test-javadoc
report:
[ERROR] Exit code: 1 -
/home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22:
error: cannot find symbol
[ERROR] import org.apache.calcite.test.CalciteAssert;
[ERROR] ^
[ERROR] symbol: class CalciteAssert
[ERROR] location: package org.apache.calcite.test
[ERROR]
/home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137:
error: package CalciteAssert does not exist
[ERROR] private CalciteAssert.AssertThat assertModel(String model) {
[ERROR] ^
[ERROR]
/home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145:
error: package CalciteAssert does not exist
[ERROR] private CalciteAssert.AssertThat assertModel(URL url) {
[ERROR] ^
[ERROR]
/home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43:
error: reference not found
[ERROR] /** Similar to {@link CalciteAssert#checkResultUnordered}, but
filters strings
[ERROR] ^
> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <[email protected]> wrote:
>
> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is JDK 8 vs
> 9 ?
>
> # JDK8 (oracle) / macOS
> $ mvn clean -DskipTests site
>
> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <[email protected]> wrote:
>
>> I just tried it again. The following fails for me (jdk9, ubuntu):
>>
>> $ git checkout c12cb4b0de
>> $ mvn clean -DskipTests site
>>
>> Constructing Javadoc information...
>> 1 error
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE [02:29
>> min]
>> [INFO] Calcite Linq4j ..................................... SKIPPED
>> [INFO] Calcite Core ....................................... SKIPPED
>> [INFO] Calcite Cassandra .................................. SKIPPED
>> [INFO] Calcite Druid ...................................... SKIPPED
>> [INFO] Calcite Elasticsearch .............................. SKIPPED
>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED
>> [INFO] Calcite Examples ................................... SKIPPED
>> [INFO] Calcite Example CSV ................................ SKIPPED
>> [INFO] Calcite Example Function ........................... SKIPPED
>> [INFO] Calcite File ....................................... SKIPPED
>> [INFO] Calcite Geode ...................................... SKIPPED
>> [INFO] Calcite MongoDB .................................... SKIPPED
>> [INFO] Calcite Pig ........................................ SKIPPED
>> [INFO] Calcite Piglet ..................................... SKIPPED
>> [INFO] Calcite Plus ....................................... SKIPPED
>> [INFO] Calcite Server ..................................... SKIPPED
>> [INFO] Calcite Spark ...................................... SKIPPED
>> [INFO] Calcite Splunk ..................................... SKIPPED
>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 02:29 min
>> [INFO] Finished at: 2018-06-22T14:53:13-07:00
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
>> project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate
>> report:
>> [ERROR] Exit code: 1 -
>> /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
>> error: package org.elasticsearch.node.internal does not exist
>> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer;
>> [ERROR] ^
>> [ERROR]
>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options @packages
>> @argfile
>> [ERROR]
>> [ERROR] Refer to the generated Javadoc files in
>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir.
>> [ERROR] -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>>
>>
>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <[email protected]> wrote:
>>>
>>> Odd, mvn clean site still works fine for me.
>>> --
>>> Michael Mior
>>> [email protected]
>>>
>>>
>>>
>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <[email protected]> a écrit :
>>>
>>>> Looks like this change broke “mvn site” (perhaps also “mvn
>>>> javadoc:test-javadoc”).
>>>>
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
>>>> project calcite: Error generating
>> maven-javadoc-plugin:3.0.1:test-aggregate
>>>> report:
>>>> [ERROR] Exit code: 1 -
>>>>
>> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
>>>> error: package org.elasticsearch.node.internal does not exist
>>>> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer;
>>>> [ERROR] ^
>>>> [ERROR]
>>>>
>>>>> On Jun 21, 2018, at 3:39 AM, [email protected] wrote:
>>>>>
>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of
>>>> ES adapter (Andrei Sereda)
>>>>>
>>>>> After discussion on dev-list Integration tests (for ES) have been
>>>> removed. They're now
>>>>> superseded by unit tests (which execute queries against a real elastic
>>>> instance)
>>>>>
>>>>> Added local file (zips-mini.json) which contains a small subset of
>>>> original zips.json
>>>>> (allows to bootstrap tests faster)
>>>>>
>>>>> Created separate ES JUnit rule which can be re-used across different
>>>> tests.
>>>>>
>>>>> Both v2 and v5 of ES adapters are supported.
>>>>>
>>>>> Close apache/calcite#716
>>>>>
>>>>>
>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo
>>>>> Commit: http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0
>>>>> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0
>>>>> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0
>>>>>
>>>>> Branch: refs/heads/master
>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d
>>>>> Parents: 37944bb
>>>>> Author: Andrei Sereda <[email protected]>
>>>>> Authored: Thu May 31 18:19:10 2018 -0400
>>>>> Committer: Michael Mior <[email protected]>
>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> .../AbstractElasticsearchTable.java | 12 +
>>>>> .../elasticsearch/ElasticsearchProject.java | 61 ++-
>>>>> elasticsearch2/pom.xml | 6 +
>>>>> .../Elasticsearch2Enumerator.java | 12 +-
>>>>> .../elasticsearch2/Elasticsearch2Schema.java | 16 +-
>>>>> .../elasticsearch2/Elasticsearch2Table.java | 9 +-
>>>>> .../ElasticSearch2AdapterTest.java | 395
>> ++++++++++++++++++
>>>>> .../elasticsearch2/EmbeddedElasticNode.java | 147 +++++++
>>>>> .../elasticsearch2/EmbeddedElasticRule.java | 97 +++++
>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++
>>>>> .../calcite/test/Elasticsearch2AdapterIT.java | 270 -------------
>>>>> .../resources/elasticsearch-zips-model.json | 50 ---
>>>>> .../src/test/resources/zips-mini.json | 149 +++++++
>>>>> elasticsearch5/pom.xml | 31 ++
>>>>> .../elasticsearch5/Elasticsearch5Schema.java | 17 +-
>>>>> .../elasticsearch5/Elasticsearch5Table.java | 11 +-
>>>>> .../ElasticSearch5AdapterTest.java | 399
>> +++++++++++++++++++
>>>>> .../elasticsearch5/EmbeddedElasticNode.java | 153 +++++++
>>>>> .../elasticsearch5/EmbeddedElasticRule.java | 98 +++++
>>>>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++
>>>>> .../calcite/test/Elasticsearch5AdapterIT.java | 270 -------------
>>>>> .../resources/elasticsearch-zips-model.json | 50 ---
>>>>> elasticsearch5/src/test/resources/log4j2.xml | 16 +
>>>>> .../src/test/resources/zips-mini.json | 149 +++++++
>>>>> pom.xml | 20 +-
>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-)
>>>>> ----------------------------------------------------------------------
>>>>>
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>> index 0980469..8cc5933 100644
>>>>> ---
>>>>
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>> +++
>>>>
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>> @@ -75,6 +75,18 @@ public abstract class AbstractElasticsearchTable
>>>> extends AbstractQueryableTable
>>>>> relOptTable, this, null);
>>>>> }
>>>>>
>>>>> + /**
>>>>> + * In ES 5.x scripted fields start with {@code params._source.foo}
>>>> while in ES2.x
>>>>> + * {@code _source.foo}. Helper method to build correct query based
>> on
>>>> runtime version of elastic.
>>>>> + *
>>>>> + * @see <a href="
>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source
>>>> variable</a>
>>>>> + * @see <a href="
>>>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html
>> ">Scripted
>>>> Fields</a>
>>>>> + */
>>>>> + protected String scriptedFieldPrefix() {
>>>>> + // this is default pattern starting 5.x
>>>>> + return "params._source";
>>>>> + }
>>>>> +
>>>>> /** Executes a "find" operation on the underlying type.
>>>>> *
>>>>> * <p>For example,
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>> index b42abd7..961c8b0 100644
>>>>> ---
>>>>
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>> +++
>>>>
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>> @@ -27,11 +27,15 @@ import
>>>> org.apache.calcite.rel.metadata.RelMetadataQuery;
>>>>> import org.apache.calcite.rel.type.RelDataType;
>>>>> import org.apache.calcite.rex.RexNode;
>>>>> import org.apache.calcite.util.Pair;
>>>>> -import org.apache.calcite.util.Util;
>>>>> +
>>>>> +import com.google.common.base.Function;
>>>>> +import com.google.common.collect.Lists;
>>>>>
>>>>> import java.util.ArrayList;
>>>>> import java.util.List;
>>>>>
>>>>> +import javax.annotation.Nullable;
>>>>> +
>>>>> /**
>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project}
>>>>> * relational expression in Elasticsearch.
>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends Project
>>>> implements ElasticsearchRel {
>>>>> implementor.visitChild(0, getInput());
>>>>>
>>>>> final List<String> inFields =
>>>>> -
>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
>>>>> +
>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
>>>>> final ElasticsearchRules.RexToElasticsearchTranslator translator =
>>>>> - new ElasticsearchRules.RexToElasticsearchTranslator(
>>>>> - (JavaTypeFactory) getCluster().getTypeFactory(),
>> inFields);
>>>>> + new ElasticsearchRules.RexToElasticsearchTranslator(
>>>>> + (JavaTypeFactory) getCluster().getTypeFactory(),
>>>> inFields);
>>>>>
>>>>> - final List<String> findItems = new ArrayList<>();
>>>>> - final List<String> scriptFieldItems = new ArrayList<>();
>>>>> + final List<String> fields = new ArrayList<>();
>>>>> + final List<String> scriptFields = new ArrayList<>();
>>>>> for (Pair<RexNode, String> pair: getNamedProjects()) {
>>>>> final String name = pair.right;
>>>>> final String expr = pair.left.accept(translator);
>>>>>
>>>>> if (expr.equals("\"" + name + "\"")) {
>>>>> - findItems.add(ElasticsearchRules.quote(name));
>>>>> + fields.add(name);
>>>>> } else if (expr.matches("\"literal\":.+")) {
>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name)
>>>>> - + ":{\"script\": "
>>>>> - + expr.split(":")[1] + "}");
>>>>> + scriptFields.add(ElasticsearchRules.quote(name)
>>>>> + + ":{\"script\": "
>>>>> + + expr.split(":")[1] + "}");
>>>>> } else {
>>>>> - scriptFieldItems.add(ElasticsearchRules.quote(name)
>>>>> - + ":{\"script\":\"params._source."
>>>>> - + expr.replaceAll("\"", "") + "\"}");
>>>>> + scriptFields.add(ElasticsearchRules.quote(name)
>>>>> + + ":{\"script\":"
>>>>> + // _source (ES2) vs params._source (ES5)
>>>>> + + "\"" +
>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
>>>>> + + expr.replaceAll("\"", "") + "\"}");
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + StringBuilder query = new StringBuilder();
>>>>> + if (scriptFields.isEmpty()) {
>>>>> + List<String> newList = Lists.transform(fields, new
>>>> Function<String, String>() {
>>>>> + @Nullable
>>>>> + @Override public String apply(@Nullable String input) {
>>>>> + return ElasticsearchRules.quote(input);
>>>>> + }
>>>>> + });
>>>>> +
>>>>> + final String findString = String.join(", ", newList);
>>>>> + query.append("\"_source\" : [").append(findString).append("]");
>>>>> + } else {
>>>>> + // if scripted fields are present, ES ignores _source attribute
>>>>> + for (String field: fields) {
>>>>> + scriptFields.add(ElasticsearchRules.quote(field) +
>>>> ":{\"script\": "
>>>>> + // _source (ES2) vs params._source (ES5)
>>>>> + + "\"" +
>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
>>>>> + + field + "\"}");
>>>>> }
>>>>> + query.append("\"script_fields\": {" + String.join(", ",
>>>> scriptFields) + "}");
>>>>> }
>>>>> - final String findString = Util.toString(findItems, "", ", ", "");
>>>>> - final String scriptFieldString = "\"script_fields\": {"
>>>>> - + Util.toString(scriptFieldItems, "", ", ", "") + "}";
>>>>> - final String fieldString = "\"_source\" : [" + findString + "]"
>>>>> - + ", " + scriptFieldString;
>>>>>
>>>>> for (String opfield : implementor.list) {
>>>>> if (opfield.startsWith("\"_source\"")) {
>>>>> implementor.list.remove(opfield);
>>>>> }
>>>>> }
>>>>> - implementor.add(fieldString);
>>>>> + implementor.add(query.toString());
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml
>>>>> ----------------------------------------------------------------------
>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml
>>>>> index f24622c..6fbee03 100644
>>>>> --- a/elasticsearch2/pom.xml
>>>>> +++ b/elasticsearch2/pom.xml
>>>>> @@ -73,6 +73,12 @@ limitations under the License.
>>>>> <version>${elasticsearch-java-driver.version}</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>> + <!-- Lang groovy dependency is needed for testing with embedded
>>>> ES (scripted fields like loc[0]) -->
>>>>> + <groupId>org.elasticsearch.module</groupId>
>>>>> + <artifactId>lang-groovy</artifactId>
>>>>> + <scope>test</scope>
>>>>> + </dependency>
>>>>> + <dependency>
>>>>> <groupId>com.carrotsearch</groupId>
>>>>> <artifactId>hppc</artifactId>
>>>>> <version>${hppc.version}</version>
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>> index 84370ab..c3d2ac0 100644
>>>>> ---
>>>>
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>> +++
>>>>
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit;
>>>>> import java.util.Date;
>>>>> import java.util.Iterator;
>>>>> import java.util.List;
>>>>> +import java.util.Locale;
>>>>> import java.util.Map;
>>>>>
>>>>> /**
>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator implements
>>>> Enumerator<Object> {
>>>>> private static Function1<SearchHit, Object[]> listGetter(
>>>>> final List<Map.Entry<String, Class>> fields) {
>>>>> return new Function1<SearchHit, Object[]>() {
>>>>> - public Object[] apply(SearchHit searchHitFields) {
>>>>> + public Object[] apply(SearchHit hit) {
>>>>> Object[] objects = new Object[fields.size()];
>>>>> for (int i = 0; i < fields.size(); i++) {
>>>>> final Map.Entry<String, Class> field = fields.get(i);
>>>>> final String name = field.getKey();
>>>>> - if (searchHitFields.fields().isEmpty()) {
>>>>> - objects[i] =
>> convert(searchHitFields.getSource().get(name),
>>>> field.getValue());
>>>>> + if (hit.fields().isEmpty()) {
>>>>> + objects[i] = convert(hit.getSource().get(name),
>>>> field.getValue());
>>>>> + } else if (hit.fields().containsKey(name)) {
>>>>> + objects[i] = convert(hit.field(name).getValue(),
>>>> field.getValue());
>>>>> } else {
>>>>> - objects[i] =
>>>> convert(searchHitFields.field(name).getValue(), field.getValue());
>>>>> + throw new IllegalStateException(
>>>>> + String.format(Locale.getDefault(), "No result for
>>>> %s", field));
>>>>> }
>>>>> }
>>>>> return objects;
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>> index 668402b..46e3fc5 100644
>>>>> ---
>>>>
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>> +++
>>>>
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>> @@ -22,6 +22,9 @@ import org.apache.calcite.schema.impl.AbstractSchema;
>>>>>
>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
>>>>>
>>>>> +import com.google.common.annotations.VisibleForTesting;
>>>>> +import com.google.common.base.Preconditions;
>>>>> +
>>>>> import com.google.common.collect.ImmutableList;
>>>>> import com.google.common.collect.ImmutableMap;
>>>>>
>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends
>>>> AbstractSchema
>>>>> }
>>>>> }
>>>>>
>>>>> + /**
>>>>> + * Allows schema to be instantiated from existing elastic search
>>>> client.
>>>>> + * This constructor is used in tests.
>>>>> + */
>>>>> + @VisibleForTesting
>>>>> + Elasticsearch2Schema(Client client, String index) {
>>>>> + this.client = Preconditions.checkNotNull(client, "client");
>>>>> + this.index = Preconditions.checkNotNull(index, "index");
>>>>> + }
>>>>> +
>>>>> @Override protected Map<String, Table> getTableMap() {
>>>>> final ImmutableMap.Builder<String, Table> builder =
>>>> ImmutableMap.builder();
>>>>>
>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends
>>>> AbstractSchema
>>>>>
>>>>> final List<DiscoveryNode> nodes =
>>>> ImmutableList.copyOf(transportClient.connectedNodes());
>>>>> if (nodes.isEmpty()) {
>>>>> - throw new RuntimeException("Cannot connect to any elasticsearch
>>>> nodes");
>>>>> + throw new IllegalStateException("Cannot connect to any
>>>> elasticsearch node: "
>>>>> + + transportNodes);
>>>>> }
>>>>>
>>>>> client = transportClient;
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>> index 636aa5f..2928835 100644
>>>>> ---
>>>>
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>> +++
>>>>
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends
>>>> AbstractElasticsearchTable {
>>>>> this.client = client;
>>>>> }
>>>>>
>>>>> + /**
>>>>> + * ES version 2.x. To access document attributes ES2 uses {@code
>>>> _source.foo} syntax.
>>>>> + */
>>>>> + @Override protected String scriptedFieldPrefix() {
>>>>> + return "_source";
>>>>> + }
>>>>> +
>>>>> @Override protected Enumerable<Object> find(String index, List<String>
>>>> ops,
>>>>> - List<Map.Entry<String, Class>> fields) {
>>>>> + List<Map.Entry<String,
>>>> Class>> fields) {
>>>>> final String dbName = index;
>>>>>
>>>>> final String queryString = "{" + Util.toString(ops, "", ", ", "") +
>>>> "}";
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>> new file mode 100644
>>>>> index 0000000..287e094
>>>>> --- /dev/null
>>>>> +++
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>> @@ -0,0 +1,395 @@
>>>>> +/*
>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> + * contributor license agreements. See the NOTICE file distributed
>> with
>>>>> + * this work for additional information regarding copyright ownership.
>>>>> + * The ASF licenses this file to you under the Apache License, Version
>>>> 2.0
>>>>> + * (the "License"); you may not use this file except in compliance
>> with
>>>>> + * the License. You may obtain a copy of the License at
>>>>> + *
>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
>>>>> + *
>>>>> + * Unless required by applicable law or agreed to in writing, software
>>>>> + * distributed under the License is distributed on an "AS IS" BASIS,
>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>>> + * See the License for the specific language governing permissions and
>>>>> + * limitations under the License.
>>>>> + */
>>>>> +package org.apache.calcite.adapter.elasticsearch2;
>>>>> +
>>>>> +import org.apache.calcite.jdbc.CalciteConnection;
>>>>> +import org.apache.calcite.schema.SchemaPlus;
>>>>> +import org.apache.calcite.schema.impl.ViewTable;
>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro;
>>>>> +import org.apache.calcite.test.CalciteAssert;
>>>>> +import org.apache.calcite.test.ElasticChecker;
>>>>> +
>>>>> +import com.google.common.io.LineProcessor;
>>>>> +import com.google.common.io.Resources;
>>>>> +
>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse;
>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder;
>>>>> +import org.elasticsearch.action.bulk.BulkResponse;
>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder;
>>>>> +import org.elasticsearch.common.xcontent.XContentFactory;
>>>>> +
>>>>> +import org.junit.BeforeClass;
>>>>> +import org.junit.ClassRule;
>>>>> +import org.junit.Test;
>>>>> +
>>>>> +import java.io.IOException;
>>>>> +import java.nio.charset.StandardCharsets;
>>>>> +import java.sql.Connection;
>>>>> +import java.sql.DriverManager;
>>>>> +import java.sql.SQLException;
>>>>> +import java.util.Arrays;
>>>>> +import java.util.Collections;
>>>>> +import java.util.Locale;
>>>>> +
>>>>> +/**
>>>>> + * Set of tests for ES adapter. Uses real instance via {@link
>>>> EmbeddedElasticRule}. Document
>>>>> + * source is local {@code zips-mini.json} file (located in the
>>>> classpath).
>>>>> + */
>>>>> +public class ElasticSearch2AdapterTest {
>>>>> +
>>>>> + @ClassRule //init once for all tests
>>>>> + public static final EmbeddedElasticRule NODE =
>>>> EmbeddedElasticRule.create();
>>>>> +
>>>>> + private static final String ZIPS = "zips";
>>>>> +
>>>>> + /**
>>>>> + * Used to create {@code zips} index and insert some data
>>>>> + */
>>>>> + @BeforeClass
>>>>> + public static void setupInstance() throws Exception {
>>>>> + // define mapping so fields are searchable (term query)
>>>>> + XContentBuilder mapping =
>>>> XContentFactory.jsonBuilder().startObject()
>>>>> + .startObject("properties")
>>>>> + .startObject("city").field("type", "string")
>>>>> + .field("index", "not_analyzed").endObject()
>>>>> + .startObject("state").field("type", "string")
>>>>> + .field("index", "not_analyzed").endObject()
>>>>> + .startObject("pop").field("type", "long").endObject()
>>>>> + .endObject()
>>>>> + .endObject();
>>>>> +
>>>>> + // create index
>>>>> + NODE.client().admin().indices()
>>>>> + .prepareCreate(ZIPS)
>>>>> + .addMapping(ZIPS, mapping)
>>>>> + .get();
>>>>> +
>>>>> + BulkRequestBuilder bulk =
>>>> NODE.client().prepareBulk().setRefresh(true);
>>>>> +
>>>>> + // load records from file
>>>>> +
>>>>
>> Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"),
>>>>> + StandardCharsets.UTF_8, new LineProcessor<Void>() {
>>>>> + @Override public boolean processLine(String line) throws
>>>> IOException {
>>>>> + line = line.replaceAll("_id", "id"); // _id is a
>>>> reserved attribute in ES
>>>>> + bulk.add(NODE.client().prepareIndex(ZIPS,
>>>> ZIPS).setSource(line));
>>>>> + return true;
>>>>> + }
>>>>> +
>>>>> + @Override public Void getResult() {
>>>>> + return null;
>>>>> + }
>>>>> + });
>>>>> +
>>>>> + if (bulk.numberOfActions() == 0) {
>>>>> + throw new IllegalStateException("No records to be indexed");
>>>>> + }
>>>>> +
>>>>> + BulkResponse response = bulk.execute().get();
>>>>> +
>>>>> + if (response.hasFailures()) {
>>>>> + throw new IllegalStateException(
>>>>> + String.format(Locale.getDefault(), "Failed to populate
>>>> %s:\n%s", NODE.httpAddress(),
>>>>> +
>>>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed)
>>>>> +
>>>>
>> .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>")));
>>>>> + }
>>>>> +
>>>>> + }
>>>>> +
>>>>> + private CalciteAssert.ConnectionFactory newConnectionFactory() {
>>>>> + return new CalciteAssert.ConnectionFactory() {
>>>>> + @Override public Connection createConnection() throws
>>>> SQLException {
>>>>> + final Connection connection =
>>>> DriverManager.getConnection("jdbc:calcite:");
>>>>> + final SchemaPlus root =
>>>> connection.unwrap(CalciteConnection.class).getRootSchema();
>>>>> +
>>>>> + root.add("elastic", new Elasticsearch2Schema(NODE.client(),
>>>> ZIPS));
>>>>> +
>>>>> + // add calcite view programmatically
>>>>> + final String viewSql = "select cast(_MAP['city'] AS
>>>> varchar(20)) AS \"city\", "
>>>>> + + " cast(_MAP['loc'][0] AS float) AS \"longitude\",\n"
>>>>> + + " cast(_MAP['loc'][1] AS float) AS \"latitude\",\n"
>>>>> + + " cast(_MAP['pop'] AS integer) AS \"pop\", "
>>>>> + + " cast(_MAP['state'] AS varchar(2)) AS \"state\", "
>>>>> + + " cast(_MAP['id'] AS varchar(5)) AS \"id\" "
>>>>> + + "from \"elastic\".\"zips\"";
>>>>> +
>>>>> + ViewTableMacro macro = ViewTable.viewMacro(root, viewSql,
>>>>> + Collections.singletonList("elastic"),
>>>> Arrays.asList("elastic", "view"), false);
>>>>> + root.add("ZIPS", macro);
>>>>> +
>>>>> + return connection;
>>>>> + }
>>>>> + };
>>>>> + }
>>>>> +
>>>>> + private CalciteAssert.AssertThat calciteAssert() {
>>>>> + return CalciteAssert.that()
>>>>> + .with(newConnectionFactory());
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Tests using calcite view
>>>>> + */
>>>>> + @Test
>>>>> + public void view() throws Exception {
>>>>> + calciteAssert()
>>>>> + .query("select * from zips where \"city\" = 'BROOKLYN'")
>>>>> + .returns("city=BROOKLYN; longitude=-73.956985;
>>>> latitude=40.646694; "
>>>>> + + "pop=111396; state=NY; id=11226\n")
>>>>> + .returnsCount(1);
>>>>> + }
>>>>> +
>>>>> + @Test
>>>>> + public void emptyResult() {
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from zips limit 0")
>>>>> + .returnsCount(0);
>>>>> +
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from \"elastic\".\"zips\" where
>>>> _MAP['Foo'] = '_MISSING_'")
>>>>> + .returnsCount(0);
>>>>> + }
>>>>> +
>>>>> + @Test
>>>>> + public void basic() throws Exception {
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from \"elastic\".\"zips\" where
>>>> _MAP['city'] = 'BROOKLYN'")
>>>>> + .returnsCount(1);
>>>>> +
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from \"elastic\".\"zips\" where"
>>>>> + + " _MAP['city'] in ('BROOKLYN', 'WASHINGTON')")
>>>>> + .returnsCount(2);
>>>>> +
>>>>> + // lower-case
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from \"elastic\".\"zips\" where "
>>>>> + + "_MAP['city'] in ('brooklyn', 'Brooklyn',
>>>> 'BROOK') ")
>>>>> + .returnsCount(0);
>>>>> +
>>>>> + // missing field
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from \"elastic\".\"zips\" where
>>>> _MAP['CITY'] = 'BROOKLYN'")
>>>>> + .returnsCount(0);
>>>>> +
>>>>> + // limit works
>>>>> + CalciteAssert.that()
>>>>> + .with(newConnectionFactory())
>>>>> + .query("select * from \"elastic\".\"zips\" limit 42")
>>>>> + .returnsCount(42);
>>>>> +
>>>>> + }
>>>>> +
>>>>> + @Test public void testSort() {
>>>>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n"
>>>>> + + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n"
>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0,
>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'),
>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5)
>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n"
>>>>> + + " ElasticsearchTableScan(table=[[elastic, zips]])";
>>>>> +
>>>>> + calciteAssert()
>>>>> + .query("select * from zips order by \"state\"")
>>>>> + .returnsCount(10)
>>>>> + .explainContains(explain);
>>>>> + }
>>>>> +
>>>>> + @Test public void testSortLimit() {
>>>>> + final String sql = "select \"state\", \"id\" from zips\n"
>>>>> + + "order by \"state\", \"id\" offset 2 rows fetch next 3
>>>> rows only";
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .returnsUnordered("state=AK; id=99801",
>>>>> + "state=AL; id=35215",
>>>>> + "state=AL; id=35401")
>>>>> + .queryContains(
>>>>> + ElasticChecker.elasticsearchChecker(
>>>>> + "\"_source\" : [\"state\", \"id\"]",
>>>>> + "\"sort\": [ {\"state\": \"asc\"},
>> {\"id\":
>>>> \"asc\"}]",
>>>>> + "\"from\": 2",
>>>>> + "\"size\": 3"));
>>>>> + }
>>>>> +
>>>>> +
>>>>> +
>>>>> + @Test public void testOffsetLimit() {
>>>>> + final String sql = "select \"state\", \"id\" from zips\n"
>>>>> + + "offset 2 fetch next 3 rows only";
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .runs()
>>>>> + .queryContains(
>>>>> + ElasticChecker.elasticsearchChecker(
>>>>> + "\"from\": 2",
>>>>> + "\"size\": 3",
>>>>> + "\"_source\" : [\"state\", \"id\"]"));
>>>>> + }
>>>>> +
>>>>> + @Test public void testLimit() {
>>>>> + final String sql = "select \"state\", \"id\" from zips\n"
>>>>> + + "fetch next 3 rows only";
>>>>> +
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .runs()
>>>>> + .queryContains(
>>>>> + ElasticChecker.elasticsearchChecker(
>>>>> + "\"size\": 3",
>>>>> + "\"_source\" : [\"state\", \"id\"]"));
>>>>> + }
>>>>> +
>>>>> + @Test public void testFilterSort() {
>>>>> + final String sql = "select * from zips\n"
>>>>> + + "where \"state\" = 'CA' and \"id\" >= '70000'\n"
>>>>> + + "order by \"state\", \"id\"";
>>>>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n"
>>>>> + + " ElasticsearchSort(sort0=[$4], sort1=[$5], dir0=[ASC],
>>>> dir1=[ASC])\n"
>>>>> + + " ElasticsearchProject(city=[CAST(ITEM($0,
>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'),
>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5)
>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n"
>>>>> + + "
>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0,
>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0, 'id')):VARCHAR(5)
>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\",
>>>> '70000'))])\n"
>>>>> + + " ElasticsearchTableScan(table=[[elastic,
>> zips]])";
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .returnsOrdered("city=LOS ANGELES; longitude=-118.258189;
>>>> latitude=34.007856; "
>>>>> + + "pop=96074; state=CA; id=90011",
>>>>> + "city=BELL GARDENS; longitude=-118.17205;
>>>> latitude=33.969177; "
>>>>> + + "pop=99568; state=CA; id=90201",
>>>>> + "city=NORWALK; longitude=-118.081767;
>>>> latitude=33.90564; "
>>>>> + + "pop=94188; state=CA; id=90650")
>>>>> + .queryContains(
>>>>> + ElasticChecker.elasticsearchChecker("\"query\" : "
>>>>> + +
>>>> "{\"constant_score\":{\"filter\":{\"bool\":"
>>>>> + +
>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}},"
>>>>> + +
>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}",
>>>>> + "\"script_fields\":
>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
>>>>> + +
>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
>>>>> + + "\"city\":{\"script\":
>>>> \"_source.city\"}, "
>>>>> + + "\"pop\":{\"script\":
>>>> \"_source.pop\"}, "
>>>>> + + "\"state\":{\"script\":
>>>> \"_source.state\"}, "
>>>>> + + "\"id\":{\"script\": \"_
>> source.id
>>>> \"}}",
>>>>> + "\"sort\": [ {\"state\": \"asc\"},
>> {\"id\":
>>>> \"asc\"}]"))
>>>>> + .explainContains(explain);
>>>>> + }
>>>>> +
>>>>> + @Test public void testFilterSortDesc() {
>>>>> + final String sql = "select * from zips\n"
>>>>> + + "where \"pop\" BETWEEN 95000 AND 100000\n"
>>>>> + + "order by \"state\" desc, \"pop\"";
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .limit(4)
>>>>> + .returnsOrdered(
>>>>> + "city=LOS ANGELES; longitude=-118.258189;
>>>> latitude=34.007856; pop=96074; state=CA; id=90011",
>>>>> + "city=BELL GARDENS; longitude=-118.17205;
>>>> latitude=33.969177; pop=99568; state=CA; id=90201");
>>>>> + }
>>>>> +
>>>>> + @Test public void testFilterRedundant() {
>>>>> + final String sql = "select * from zips\n"
>>>>> + + "where \"state\" > 'CA' and \"state\" < 'AZ' and
>>>> \"state\" = 'OK'";
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .runs()
>>>>> + .queryContains(
>>>>> + ElasticChecker.elasticsearchChecker(""
>>>>> + + "\"query\" :
>>>> {\"constant_score\":{\"filter\":{\"bool\":"
>>>>> + +
>>>> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}",
>>>>> + "\"script_fields\":
>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
>>>>> + +
>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
>>>>> + + "\"city\":{\"script\":
>>>> \"_source.city\"}, "
>>>>> + + "\"pop\":{\"script\":
>>>> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, "
>>>>> + + "\"id\":{\"script\":
>>>> \"_source.id\"}}"
>>>>> + ));
>>>>> + }
>>>>> +
>>>>> + @Test public void testInPlan() {
>>>>> + final String[] searches = {
>>>>> + "\"query\" :
>>>> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":"
>>>>> + +
>>>>
>> "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":"
>>>>> + + "{\"pop\":99568}}]}}]}}}}",
>>>>> + "\"script_fields\":
>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
>>>>> + + "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
>>>>> + + "\"city\":{\"script\": \"_source.city\"}, "
>>>>> + + "\"pop\":{\"script\": \"_source.pop\"}, "
>>>>> + + "\"state\":{\"script\": \"_source.state\"}, "
>>>>> + + "\"id\":{\"script\": \"_source.id\"}}"
>>>>> + };
>>>>> +
>>>>> + calciteAssert()
>>>>> + .query("select * from zips where \"pop\" in (96074,
>> 99568)")
>>>>> + .returnsUnordered(
>>>>> + "city=BELL GARDENS; longitude=-118.17205;
>>>> latitude=33.969177; pop=99568; state=CA; id=90201",
>>>>> + "city=LOS ANGELES; longitude=-118.258189;
>>>> latitude=34.007856; pop=96074; state=CA; id=90011"
>>>>> + )
>>>>> +
>>>> .queryContains(ElasticChecker.elasticsearchChecker(searches));
>>>>> + }
>>>>> +
>>>>> + @Test public void testZips() {
>>>>> + calciteAssert()
>>>>> + .query("select \"state\", \"city\" from zips")
>>>>> + .returnsCount(10);
>>>>> + }
>>>>> +
>>>>> + @Test public void testProject() {
>>>>> + final String sql = "select \"state\", \"city\", 0 as \"zero\"\n"
>>>>> + + "from zips\n"
>>>>> + + "order by \"state\", \"city\"";
>>>>> +
>>>>> + calciteAssert()
>>>>> + .query(sql)
>>>>> + .limit(2)
>>>>> + .returnsUnordered("state=AK; city=ANCHORAGE; zero=0",
>>>>> + "state=AK; city=FAIRBANKS; zero=0")
>>>>> + .queryContains(
>>>>> +
>>>> ElasticChecker.elasticsearchChecker("\"script_fields\": "
>>>>> + + "{\"zero\":{\"script\": \"0\"},
>> "
>>>>> + + "\"state\":{\"script\":
>>>> \"_source.state\"}, "
>>>>> + + "\"city\":{\"script\":
>>>> \"_source.city\"}}",
>>>>> + "\"sort\": [ {\"state\": \"asc\"},
>>>> {\"city\": \"asc\"}]"));
>>>>> + }
>>>>> +
>>>>> + @Test public void testFilter() {
>>>>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n"
>>>>> + + " ElasticsearchProject(state=[CAST(ITEM($0,
>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"], city=[CAST(ITEM($0, 'city')):VARCHAR(20)
>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n"
>>>>> + + " ElasticsearchFilter(condition=[=(CAST(ITEM($0,
>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\", 'CA')])\n"
>>>>> + + " ElasticsearchTableScan(table=[[elastic, zips]])";
>>>>> + calciteAssert()
>>>>> + .query("select \"state\", \"city\" from zips where
>>>> \"state\" = 'CA'")
>>>>> + .limit(3)
>>>>> + .returnsUnordered("state=CA; city=BELL GARDENS",
>>>>> + "state=CA; city=LOS ANGELES",
>>>>> + "state=CA; city=NORWALK")
>>>>> + .explainContains(explain);
>>>>> + }
>>>>> +
>>>>> + @Test public void testFilterReversed() {
>>>>> + calciteAssert()
>>>>> + .query("select \"state\", \"city\" from zips where 'WI' <
>>>> \"state\" order by \"city\"")
>>>>> + .limit(2)
>>>>> + .returnsUnordered("state=WV; city=BECKLEY",
>>>>> + "state=WY; city=CHEYENNE");
>>>>> + calciteAssert()
>>>>> + .query("select \"state\", \"city\" from zips where \"state\"
>>>>> 'WI' order by \"city\"")
>>>>> + .limit(2)
>>>>> + .returnsUnordered("state=WV; city=BECKLEY",
>>>>> + "state=WY; city=CHEYENNE");
>>>>> + }
>>>>> +
>>>>> +}
>>>>> +
>>>>> +// End ElasticSearch2AdapterTest.java
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
>>>>> new file mode 100644
>>>>> index 0000000..4474add
>>>>> --- /dev/null
>>>>> +++
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
>>>>> @@ -0,0 +1,147 @@
>>>>> +/*
>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> + * contributor license agreements. See the NOTICE file distributed
>> with
>>>>> + * this work for additional information regarding copyright ownership.
>>>>> + * The ASF licenses this file to you under the Apache License, Version
>>>> 2.0
>>>>> + * (the "License"); you may not use this file except in compliance
>> with
>>>>> + * the License. You may obtain a copy of the License at
>>>>> + *
>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
>>>>> + *
>>>>> + * Unless required by applicable law or agreed to in writing, software
>>>>> + * distributed under the License is distributed on an "AS IS" BASIS,
>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>>> + * See the License for the specific language governing permissions and
>>>>> + * limitations under the License.
>>>>> + */
>>>>> +package org.apache.calcite.adapter.elasticsearch2;
>>>>> +
>>>>> +import com.google.common.base.Preconditions;
>>>>> +import com.google.common.io.Files;
>>>>> +
>>>>> +import org.elasticsearch.Version;
>>>>> +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo;
>>>>> +import org.elasticsearch.action.admin.cluster.node.info
>>>> .NodesInfoResponse;
>>>>> +import org.elasticsearch.client.Client;
>>>>> +import org.elasticsearch.common.settings.Settings;
>>>>> +import org.elasticsearch.common.transport.TransportAddress;
>>>>> +import org.elasticsearch.node.Node;
>>>>> +import org.elasticsearch.node.internal.InternalSettingsPreparer;
>>>>> +import org.elasticsearch.plugins.Plugin;
>>>>> +import org.elasticsearch.script.groovy.GroovyPlugin;
>>>>> +
>>>>> +import java.io.File;
>>>>> +import java.util.Arrays;
>>>>> +import java.util.Collection;
>>>>> +import java.util.Collections;
>>>>> +
>>>>> +/**
>>>>> + * Represents a single elastic search node which can run embedded in a
>>>> java application.
>>>>> + * Intended for unit and integration tests. Settings and plugins are
>>>> crafted for Calcite.
>>>>> + */
>>>>> +class EmbeddedElasticNode implements AutoCloseable {
>>>>> +
>>>>> + private final LocalNode node;
>>>>> + private volatile boolean isStarted;
>>>>> +
>>>>> + private EmbeddedElasticNode(LocalNode node) {
>>>>> + this.node = Preconditions.checkNotNull(node, "node");
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Having separate class to expose (protected) constructor which
>>>> allows to install
>>>>> + * different plugins. In our case it is {@code GroovyPlugin} for
>>>> scripted fields like
>>>>> + * {@code loc[0]} or {@code loc[1]['foo']}.
>>>>> + */
>>>>> + private static class LocalNode extends Node {
>>>>> + private LocalNode(Settings settings, Collection<Class<? extends
>>>> Plugin>> classpathPlugins) {
>>>>> + super(InternalSettingsPreparer.prepareEnvironment(settings,
>> null),
>>>>> + Version.CURRENT,
>>>>> + classpathPlugins);
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Creates an instance with existing settings
>>>>> + */
>>>>> + private static EmbeddedElasticNode create(Settings settings) {
>>>>> + // ensure GroovyPlugin is installed or otherwise scripted fields
>>>> would not work
>>>>> + LocalNode node = new LocalNode(settings,
>>>> Collections.singleton(GroovyPlugin.class));
>>>>> + return new EmbeddedElasticNode(node);
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Creates elastic node as single member of a cluster. Node will not
>>>> be started
>>>>> + * unless {@link #start()} is explicitly called.
>>>>> + */
>>>>> + public static EmbeddedElasticNode create() {
>>>>> + File data = Files.createTempDir();
>>>>> + data.deleteOnExit();
>>>>> + File home = Files.createTempDir();
>>>>> + home.deleteOnExit();
>>>>> +
>>>>> + Settings settings = Settings.builder()
>>>>> + .put("node.name", "fake-elastic")
>>>>> + .put("path.home", home.getAbsolutePath())
>>>>> + .put("path.data", data.getAbsolutePath())
>>>>> + .put("script.inline", true) // requires GroovyPlugin
>>>>> + .put("script.indexed", true) // requires GroovyPlugin
>>>>> + .put("cluster.routing.allocation.disk.threshold_enabled",
>>>> false)
>>>>> + .put("node.local", true)
>>>>> + .put("node.data", true)
>>>>> + .put("network.host", "localhost")
>>>>> + .build();
>>>>> +
>>>>> + return create(settings);
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Starts current node
>>>>> + */
>>>>> + public void start() {
>>>>> + Preconditions.checkState(!isStarted, "already started");
>>>>> + node.start();
>>>>> + this.isStarted = true;
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Returns current address to connect to with HTTP client.
>>>>> + */
>>>>> + public TransportAddress httpAddress() {
>>>>> + Preconditions.checkState(isStarted, "node is not started");
>>>>> +
>>>>> + NodesInfoResponse response =
>>>> client().admin().cluster().prepareNodesInfo()
>>>>> + .execute().actionGet();
>>>>> + if (response.getNodes().length != 1) {
>>>>> + throw new IllegalStateException("Expected single node but got "
>>>>> + + response.getNodes().length);
>>>>> + }
>>>>> + NodeInfo node = response.getNodes()[0];
>>>>> + return node.getHttp().address().boundAddresses()[0];
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Exposes elastic
>>>>> + * <a href="
>>>>
>> https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html
>> ">transport
>>>> client</a>
>>>>> + *
>>>>> + * (use of HTTP client is preferred).
>>>>> + */
>>>>> + public Client client() {
>>>>> + Preconditions.checkState(isStarted, "node is not started");
>>>>> + return node.client();
>>>>> + }
>>>>> +
>>>>> + @Override public void close() throws Exception {
>>>>> + node.close();
>>>>> + // cleanup data dirs
>>>>> + File data = new File(node.settings().get("path.data"));
>>>>> + File home = new File(node.settings().get("path.home"));
>>>>> + for (File file: Arrays.asList(data, home)) {
>>>>> + if (file.exists()) {
>>>>> + file.delete();
>>>>> + }
>>>>> + }
>>>>> + }
>>>>> +}
>>>>> +
>>>>> +// End EmbeddedElasticNode.java
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java
>>>>> new file mode 100644
>>>>> index 0000000..a633078
>>>>> --- /dev/null
>>>>> +++
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java
>>>>> @@ -0,0 +1,97 @@
>>>>> +/*
>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> + * contributor license agreements. See the NOTICE file distributed
>> with
>>>>> + * this work for additional information regarding copyright ownership.
>>>>> + * The ASF licenses this file to you under the Apache License, Version
>>>> 2.0
>>>>> + * (the "License"); you may not use this file except in compliance
>> with
>>>>> + * the License. You may obtain a copy of the License at
>>>>> + *
>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
>>>>> + *
>>>>> + * Unless required by applicable law or agreed to in writing, software
>>>>> + * distributed under the License is distributed on an "AS IS" BASIS,
>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>>> + * See the License for the specific language governing permissions and
>>>>> + * limitations under the License.
>>>>> + */
>>>>> +package org.apache.calcite.adapter.elasticsearch2;
>>>>> +
>>>>> +import com.google.common.base.Preconditions;
>>>>> +
>>>>> +import org.elasticsearch.client.Client;
>>>>> +import org.elasticsearch.common.transport.TransportAddress;
>>>>> +import org.junit.rules.ExternalResource;
>>>>> +
>>>>> +/**
>>>>> + * Used to initialize a single elastic node. For performance reasons
>>>> (node startup costs),
>>>>> + * same instance is usually shared across multiple tests.
>>>>> + *
>>>>> + * This rule should be used as follows:
>>>>> + * <pre>
>>>>> + * {@code
>>>>> + *
>>>>> + * public class MyTest {
>>>>> + * @literal @ClassRule
>>>>> + * public static final ElasticSearchRule RULE =
>>>> ElasticSearchRule.create();
>>>>> + *
>>>>> + * @literal @BeforeClass
>>>>> + * public void setup() {
>>>>> + * // ... populate instance
>>>>> + * }
>>>>> + *
>>>>> + * @literal @Test
>>>>> + * public void myTest() {
>>>>> + * TransportAddress address = RULE.httpAddress();
>>>>> + * // ....
>>>>> + * }
>>>>> + * }
>>>>> + * }
>>>>> + * </pre>
>>>>> + *
>>>>> + * @see ExternalResource
>>>>> + */
>>>>> +class EmbeddedElasticRule extends ExternalResource {
>>>>> +
>>>>> + private final EmbeddedElasticNode node;
>>>>> +
>>>>> + private EmbeddedElasticRule(EmbeddedElasticNode resource) {
>>>>> + this.node = Preconditions.checkNotNull(resource, "resource");
>>>>> + }
>>>>> +
>>>>> + @Override protected void before() throws Throwable {
>>>>> + node.start();
>>>>> + }
>>>>> +
>>>>> + @Override protected void after() {
>>>>> + try {
>>>>> + node.close();
>>>>> + } catch (Exception e) {
>>>>> + throw new RuntimeException(e);
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Factory method to create this rule.
>>>>> + */
>>>>> + public static EmbeddedElasticRule create() {
>>>>> + return new EmbeddedElasticRule(EmbeddedElasticNode.create());
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * Exposes current ES transport client.
>>>>> + */
>>>>> + Client client() {
>>>>> + return node.client();
>>>>> + }
>>>>> +
>>>>> + /**
>>>>> + * HTTP address for rest clients (can be ES native or any other).
>>>>> + */
>>>>> + TransportAddress httpAddress() {
>>>>> + return node.httpAddress();
>>>>> + }
>>>>> +
>>>>> +
>>>>> +}
>>>>> +
>>>>> +// End EmbeddedElasticRule.java
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java
>>>>> new file mode 100644
>>>>> index 0000000..21fc491
>>>>> --- /dev/null
>>>>> +++
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java
>>>>> @@ -0,0 +1,49 @@
>>>>> +/*
>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> + * contributor license agreements. See the NOTICE file distributed
>> with
>>>>> + * this work for additional information regarding copyright ownership.
>>>>> + * The ASF licenses this file to you under the Apache License, Version
>>>> 2.0
>>>>> + * (the "License"); you may not use this file except in compliance
>> with
>>>>> + * the License. You may obtain a copy of the License at
>>>>> + *
>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
>>>>> + *
>>>>> + * Unless required by applicable law or agreed to in writing, software
>>>>> + * distributed under the License is distributed on an "AS IS" BASIS,
>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>>> + * See the License for the specific language governing permissions and
>>>>> + * limitations under the License.
>>>>> + */
>>>>> +package org.apache.calcite.test;
>>>>> +
>>>>> +import com.google.common.base.Function;
>>>>> +
>>>>> +import java.util.List;
>>>>> +
>>>>> +import javax.annotation.Nullable;
>>>>> +
>>>>> +/**
>>>>> + * Internal util methods for ElasticSearch tests
>>>>> + */
>>>>> +public class ElasticChecker {
>>>>> +
>>>>> + private ElasticChecker() {}
>>>>> +
>>>>> +
>>>>> + /** Returns a function that checks that a particular Elasticsearch
>>>> pipeline is
>>>>> + * generated to implement a query. */
>>>>> + public static Function<List, Void> elasticsearchChecker(final
>>>> String... strings) {
>>>>> + return new Function<List, Void>() {
>>>>> + @Nullable
>>>>> + @Override public Void apply(@Nullable List actual) {
>>>>> + Object[] actualArray = actual == null || actual.isEmpty() ?
>> null
>>>>> + : ((List) actual.get(0)).toArray();
>>>>> + CalciteAssert.assertArrayEqual("expected Elasticsearch query
>>>> not found", strings,
>>>>> + actualArray);
>>>>> + return null;
>>>>> + }
>>>>> + };
>>>>> + }
>>>>> +}
>>>>> +
>>>>> +// End ElasticChecker.java
>>>>>
>>>>>
>>>>
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java
>>>>> ----------------------------------------------------------------------
>>>>> diff --git
>>>>
>> a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java
>>>>
>> b/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java
>>>>> deleted file mode 100644
>>>>> index 4e0c2b6..0000000
>>>>> ---
>>>>
>> a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java
>>>>> +++ /dev/null
>>>>> @@ -1,270 +0,0 @@
>>>>> -/*
>>>>> - * Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> - * contributor license agreements. See the NOTICE file distributed
>> with
>>>>> - * this work for additional information regarding copyright ownership.
>>>>> - * The ASF licenses this file to you under the Apache License, Version
>>>> 2.0
>>>>> - * (the "License"); you may not use this file except in compliance
>> with
>>>>> - * the License. You may obtain a copy of the License at
>>>>> - *
>>>>> - * http://www.apache.org/licenses/LICENSE-2.0
>>>>> - *
>>>>> - * Unless required by applicable law or agreed to in writing, software
>>>>> - * distributed under the License is distributed on an "AS IS" BASIS,
>>>>> - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>>> - * See the License for the specific language governing permissions and
>>>>> - * limitations under the License.
>>>>> - */
>>>>> -package org.apache.calcite.test;
>>>>> -
>>>>> -import org.apache.calcite.util.Util;
>>>>> -
>>>>> -import com.google.common.base.Function;
>>>>> -import com.google.common.collect.ImmutableMap;
>>>>> -
>>>>> -import org.junit.Test;
>>>>> -
>>>>> -import java.util.List;
>>>>> -import javax.annotation.Nullable;
>>>>> -
>>>>> -/**
>>>>> - * Tests for the {@code org.apache.calcite.adapter.elasticsearch2}
>>>> package.
>>>>> - *
>>>>> - * <p>Before calling this test, you need to populate Elasticsearch, as
>>>> follows:
>>>>> - *
>>>>> - * <blockquote><code>
>>>>> - * git clone https://github.com/vlsi/calcite-test-dataset<br>
>>>>> - * cd calcite-test-dataset<br>
>>>>> - * mvn install
>>>>> - * </code></blockquote>
>>>>> - *
>>>>> - * <p>This will create a virtual machine with Elastic
