Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is JDK 8 vs 9 ?
# JDK8 (oracle) / macOS $ mvn clean -DskipTests site On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <[email protected]> wrote: > I just tried it again. The following fails for me (jdk9, ubuntu): > > $ git checkout c12cb4b0de > $ mvn clean -DskipTests site > > Constructing Javadoc information... > 1 error > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE [02:29 > min] > [INFO] Calcite Linq4j ..................................... SKIPPED > [INFO] Calcite Core ....................................... SKIPPED > [INFO] Calcite Cassandra .................................. SKIPPED > [INFO] Calcite Druid ...................................... SKIPPED > [INFO] Calcite Elasticsearch .............................. SKIPPED > [INFO] Calcite Elasticsearch5 ............................. SKIPPED > [INFO] Calcite Examples ................................... SKIPPED > [INFO] Calcite Example CSV ................................ SKIPPED > [INFO] Calcite Example Function ........................... SKIPPED > [INFO] Calcite File ....................................... SKIPPED > [INFO] Calcite Geode ...................................... SKIPPED > [INFO] Calcite MongoDB .................................... SKIPPED > [INFO] Calcite Pig ........................................ SKIPPED > [INFO] Calcite Piglet ..................................... SKIPPED > [INFO] Calcite Plus ....................................... SKIPPED > [INFO] Calcite Server ..................................... SKIPPED > [INFO] Calcite Spark ...................................... SKIPPED > [INFO] Calcite Splunk ..................................... SKIPPED > [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 02:29 min > [INFO] Finished at: 2018-06-22T14:53:13-07:00 > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate > report: > [ERROR] Exit code: 1 - > /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > error: package org.elasticsearch.node.internal does not exist > [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; > [ERROR] ^ > [ERROR] > [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options @packages > @argfile > [ERROR] > [ERROR] Refer to the generated Javadoc files in > '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir. > [ERROR] -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > > > > On Jun 22, 2018, at 2:41 PM, Michael Mior <[email protected]> wrote: > > > > Odd, mvn clean site still works fine for me. > > -- > > Michael Mior > > [email protected] > > > > > > > > Le ven. 22 juin 2018 à 17:12, Julian Hyde <[email protected]> a écrit : > > > >> Looks like this change broke “mvn site” (perhaps also “mvn > >> javadoc:test-javadoc”). > >> > >> [ERROR] Failed to execute goal > >> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on > >> project calcite: Error generating > maven-javadoc-plugin:3.0.1:test-aggregate > >> report: > >> [ERROR] Exit code: 1 - > >> > /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: > >> error: package org.elasticsearch.node.internal does not exist > >> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; > >> [ERROR] ^ > >> [ERROR] > >> > >>> On Jun 21, 2018, at 3:39 AM, [email protected] wrote: > >>> > >>> [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of > >> ES adapter (Andrei Sereda) > >>> > >>> After discussion on dev-list Integration tests (for ES) have been > >> removed. They're now > >>> superseded by unit tests (which execute queries against a real elastic > >> instance) > >>> > >>> Added local file (zips-mini.json) which contains a small subset of > >> original zips.json > >>> (allows to bootstrap tests faster) > >>> > >>> Created separate ES JUnit rule which can be re-used across different > >> tests. > >>> > >>> Both v2 and v5 of ES adapters are supported. > >>> > >>> Close apache/calcite#716 > >>> > >>> > >>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo > >>> Commit: http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0 > >>> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0 > >>> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0 > >>> > >>> Branch: refs/heads/master > >>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d > >>> Parents: 37944bb > >>> Author: Andrei Sereda <[email protected]> > >>> Authored: Thu May 31 18:19:10 2018 -0400 > >>> Committer: Michael Mior <[email protected]> > >>> Committed: Thu Jun 21 06:38:50 2018 -0400 > >>> > >>> ---------------------------------------------------------------------- > >>> .../AbstractElasticsearchTable.java | 12 + > >>> .../elasticsearch/ElasticsearchProject.java | 61 ++- > >>> elasticsearch2/pom.xml | 6 + > >>> .../Elasticsearch2Enumerator.java | 12 +- > >>> .../elasticsearch2/Elasticsearch2Schema.java | 16 +- > >>> .../elasticsearch2/Elasticsearch2Table.java | 9 +- > >>> .../ElasticSearch2AdapterTest.java | 395 > ++++++++++++++++++ > >>> .../elasticsearch2/EmbeddedElasticNode.java | 147 +++++++ > >>> .../elasticsearch2/EmbeddedElasticRule.java | 97 +++++ > >>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ > >>> .../calcite/test/Elasticsearch2AdapterIT.java | 270 ------------- > >>> .../resources/elasticsearch-zips-model.json | 50 --- > >>> .../src/test/resources/zips-mini.json | 149 +++++++ > >>> elasticsearch5/pom.xml | 31 ++ > >>> .../elasticsearch5/Elasticsearch5Schema.java | 17 +- > >>> .../elasticsearch5/Elasticsearch5Table.java | 11 +- > >>> .../ElasticSearch5AdapterTest.java | 399 > +++++++++++++++++++ > >>> .../elasticsearch5/EmbeddedElasticNode.java | 153 +++++++ > >>> .../elasticsearch5/EmbeddedElasticRule.java | 98 +++++ > >>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ > >>> .../calcite/test/Elasticsearch5AdapterIT.java | 270 ------------- > >>> .../resources/elasticsearch-zips-model.json | 50 --- > >>> elasticsearch5/src/test/resources/log4j2.xml | 16 + > >>> .../src/test/resources/zips-mini.json | 149 +++++++ > >>> pom.xml | 20 +- > >>> 25 files changed, 1866 insertions(+), 670 deletions(-) > >>> ---------------------------------------------------------------------- > >>> > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>> index 0980469..8cc5933 100644 > >>> --- > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>> +++ > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java > >>> @@ -75,6 +75,18 @@ public abstract class AbstractElasticsearchTable > >> extends AbstractQueryableTable > >>> relOptTable, this, null); > >>> } > >>> > >>> + /** > >>> + * In ES 5.x scripted fields start with {@code params._source.foo} > >> while in ES2.x > >>> + * {@code _source.foo}. Helper method to build correct query based > on > >> runtime version of elastic. > >>> + * > >>> + * @see <a href=" > >> https://github.com/elastic/elasticsearch/issues/20068";>_source > >> variable</a> > >>> + * @see <a href=" > >> > https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html > ">Scripted > >> Fields</a> > >>> + */ > >>> + protected String scriptedFieldPrefix() { > >>> + // this is default pattern starting 5.x > >>> + return "params._source"; > >>> + } > >>> + > >>> /** Executes a "find" operation on the underlying type. > >>> * > >>> * <p>For example, > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>> index b42abd7..961c8b0 100644 > >>> --- > >> > a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>> +++ > >> > b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java > >>> @@ -27,11 +27,15 @@ import > >> org.apache.calcite.rel.metadata.RelMetadataQuery; > >>> import org.apache.calcite.rel.type.RelDataType; > >>> import org.apache.calcite.rex.RexNode; > >>> import org.apache.calcite.util.Pair; > >>> -import org.apache.calcite.util.Util; > >>> + > >>> +import com.google.common.base.Function; > >>> +import com.google.common.collect.Lists; > >>> > >>> import java.util.ArrayList; > >>> import java.util.List; > >>> > >>> +import javax.annotation.Nullable; > >>> + > >>> /** > >>> * Implementation of {@link org.apache.calcite.rel.core.Project} > >>> * relational expression in Elasticsearch. > >>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends Project > >> implements ElasticsearchRel { > >>> implementor.visitChild(0, getInput()); > >>> > >>> final List<String> inFields = > >>> - > >> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); > >>> + > >> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); > >>> final ElasticsearchRules.RexToElasticsearchTranslator translator = > >>> - new ElasticsearchRules.RexToElasticsearchTranslator( > >>> - (JavaTypeFactory) getCluster().getTypeFactory(), > inFields); > >>> + new ElasticsearchRules.RexToElasticsearchTranslator( > >>> + (JavaTypeFactory) getCluster().getTypeFactory(), > >> inFields); > >>> > >>> - final List<String> findItems = new ArrayList<>(); > >>> - final List<String> scriptFieldItems = new ArrayList<>(); > >>> + final List<String> fields = new ArrayList<>(); > >>> + final List<String> scriptFields = new ArrayList<>(); > >>> for (Pair<RexNode, String> pair: getNamedProjects()) { > >>> final String name = pair.right; > >>> final String expr = pair.left.accept(translator); > >>> > >>> if (expr.equals("\"" + name + "\"")) { > >>> - findItems.add(ElasticsearchRules.quote(name)); > >>> + fields.add(name); > >>> } else if (expr.matches("\"literal\":.+")) { > >>> - scriptFieldItems.add(ElasticsearchRules.quote(name) > >>> - + ":{\"script\": " > >>> - + expr.split(":")[1] + "}"); > >>> + scriptFields.add(ElasticsearchRules.quote(name) > >>> + + ":{\"script\": " > >>> + + expr.split(":")[1] + "}"); > >>> } else { > >>> - scriptFieldItems.add(ElasticsearchRules.quote(name) > >>> - + ":{\"script\":\"params._source." > >>> - + expr.replaceAll("\"", "") + "\"}"); > >>> + scriptFields.add(ElasticsearchRules.quote(name) > >>> + + ":{\"script\":" > >>> + // _source (ES2) vs params._source (ES5) > >>> + + "\"" + > >> implementor.elasticsearchTable.scriptedFieldPrefix() + "." > >>> + + expr.replaceAll("\"", "") + "\"}"); > >>> + } > >>> + } > >>> + > >>> + StringBuilder query = new StringBuilder(); > >>> + if (scriptFields.isEmpty()) { > >>> + List<String> newList = Lists.transform(fields, new > >> Function<String, String>() { > >>> + @Nullable > >>> + @Override public String apply(@Nullable String input) { > >>> + return ElasticsearchRules.quote(input); > >>> + } > >>> + }); > >>> + > >>> + final String findString = String.join(", ", newList); > >>> + query.append("\"_source\" : [").append(findString).append("]"); > >>> + } else { > >>> + // if scripted fields are present, ES ignores _source attribute > >>> + for (String field: fields) { > >>> + scriptFields.add(ElasticsearchRules.quote(field) + > >> ":{\"script\": " > >>> + // _source (ES2) vs params._source (ES5) > >>> + + "\"" + > >> implementor.elasticsearchTable.scriptedFieldPrefix() + "." > >>> + + field + "\"}"); > >>> } > >>> + query.append("\"script_fields\": {" + String.join(", ", > >> scriptFields) + "}"); > >>> } > >>> - final String findString = Util.toString(findItems, "", ", ", ""); > >>> - final String scriptFieldString = "\"script_fields\": {" > >>> - + Util.toString(scriptFieldItems, "", ", ", "") + "}"; > >>> - final String fieldString = "\"_source\" : [" + findString + "]" > >>> - + ", " + scriptFieldString; > >>> > >>> for (String opfield : implementor.list) { > >>> if (opfield.startsWith("\"_source\"")) { > >>> implementor.list.remove(opfield); > >>> } > >>> } > >>> - implementor.add(fieldString); > >>> + implementor.add(query.toString()); > >>> } > >>> } > >>> > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml > >>> ---------------------------------------------------------------------- > >>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml > >>> index f24622c..6fbee03 100644 > >>> --- a/elasticsearch2/pom.xml > >>> +++ b/elasticsearch2/pom.xml > >>> @@ -73,6 +73,12 @@ limitations under the License. > >>> <version>${elasticsearch-java-driver.version}</version> > >>> </dependency> > >>> <dependency> > >>> + <!-- Lang groovy dependency is needed for testing with embedded > >> ES (scripted fields like loc[0]) --> > >>> + <groupId>org.elasticsearch.module</groupId> > >>> + <artifactId>lang-groovy</artifactId> > >>> + <scope>test</scope> > >>> + </dependency> > >>> + <dependency> > >>> <groupId>com.carrotsearch</groupId> > >>> <artifactId>hppc</artifactId> > >>> <version>${hppc.version}</version> > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>> index 84370ab..c3d2ac0 100644 > >>> --- > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>> +++ > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java > >>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit; > >>> import java.util.Date; > >>> import java.util.Iterator; > >>> import java.util.List; > >>> +import java.util.Locale; > >>> import java.util.Map; > >>> > >>> /** > >>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator implements > >> Enumerator<Object> { > >>> private static Function1<SearchHit, Object[]> listGetter( > >>> final List<Map.Entry<String, Class>> fields) { > >>> return new Function1<SearchHit, Object[]>() { > >>> - public Object[] apply(SearchHit searchHitFields) { > >>> + public Object[] apply(SearchHit hit) { > >>> Object[] objects = new Object[fields.size()]; > >>> for (int i = 0; i < fields.size(); i++) { > >>> final Map.Entry<String, Class> field = fields.get(i); > >>> final String name = field.getKey(); > >>> - if (searchHitFields.fields().isEmpty()) { > >>> - objects[i] = > convert(searchHitFields.getSource().get(name), > >> field.getValue()); > >>> + if (hit.fields().isEmpty()) { > >>> + objects[i] = convert(hit.getSource().get(name), > >> field.getValue()); > >>> + } else if (hit.fields().containsKey(name)) { > >>> + objects[i] = convert(hit.field(name).getValue(), > >> field.getValue()); > >>> } else { > >>> - objects[i] = > >> convert(searchHitFields.field(name).getValue(), field.getValue()); > >>> + throw new IllegalStateException( > >>> + String.format(Locale.getDefault(), "No result for > >> %s", field)); > >>> } > >>> } > >>> return objects; > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>> index 668402b..46e3fc5 100644 > >>> --- > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>> +++ > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java > >>> @@ -22,6 +22,9 @@ import org.apache.calcite.schema.impl.AbstractSchema; > >>> > >>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor; > >>> > >>> +import com.google.common.annotations.VisibleForTesting; > >>> +import com.google.common.base.Preconditions; > >>> + > >>> import com.google.common.collect.ImmutableList; > >>> import com.google.common.collect.ImmutableMap; > >>> > >>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends > >> AbstractSchema > >>> } > >>> } > >>> > >>> + /** > >>> + * Allows schema to be instantiated from existing elastic search > >> client. > >>> + * This constructor is used in tests. > >>> + */ > >>> + @VisibleForTesting > >>> + Elasticsearch2Schema(Client client, String index) { > >>> + this.client = Preconditions.checkNotNull(client, "client"); > >>> + this.index = Preconditions.checkNotNull(index, "index"); > >>> + } > >>> + > >>> @Override protected Map<String, Table> getTableMap() { > >>> final ImmutableMap.Builder<String, Table> builder = > >> ImmutableMap.builder(); > >>> > >>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends > >> AbstractSchema > >>> > >>> final List<DiscoveryNode> nodes = > >> ImmutableList.copyOf(transportClient.connectedNodes()); > >>> if (nodes.isEmpty()) { > >>> - throw new RuntimeException("Cannot connect to any elasticsearch > >> nodes"); > >>> + throw new IllegalStateException("Cannot connect to any > >> elasticsearch node: " > >>> + + transportNodes); > >>> } > >>> > >>> client = transportClient; > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>> index 636aa5f..2928835 100644 > >>> --- > >> > a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>> +++ > >> > b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java > >>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends > >> AbstractElasticsearchTable { > >>> this.client = client; > >>> } > >>> > >>> + /** > >>> + * ES version 2.x. To access document attributes ES2 uses {@code > >> _source.foo} syntax. > >>> + */ > >>> + @Override protected String scriptedFieldPrefix() { > >>> + return "_source"; > >>> + } > >>> + > >>> @Override protected Enumerable<Object> find(String index, List<String> > >> ops, > >>> - List<Map.Entry<String, Class>> fields) { > >>> + List<Map.Entry<String, > >> Class>> fields) { > >>> final String dbName = index; > >>> > >>> final String queryString = "{" + Util.toString(ops, "", ", ", "") + > >> "}"; > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>> new file mode 100644 > >>> index 0000000..287e094 > >>> --- /dev/null > >>> +++ > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java > >>> @@ -0,0 +1,395 @@ > >>> +/* > >>> + * Licensed to the Apache Software Foundation (ASF) under one or more > >>> + * contributor license agreements. See the NOTICE file distributed > with > >>> + * this work for additional information regarding copyright ownership. > >>> + * The ASF licenses this file to you under the Apache License, Version > >> 2.0 > >>> + * (the "License"); you may not use this file except in compliance > with > >>> + * the License. You may obtain a copy of the License at > >>> + * > >>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>> + * > >>> + * Unless required by applicable law or agreed to in writing, software > >>> + * distributed under the License is distributed on an "AS IS" BASIS, > >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >> implied. > >>> + * See the License for the specific language governing permissions and > >>> + * limitations under the License. > >>> + */ > >>> +package org.apache.calcite.adapter.elasticsearch2; > >>> + > >>> +import org.apache.calcite.jdbc.CalciteConnection; > >>> +import org.apache.calcite.schema.SchemaPlus; > >>> +import org.apache.calcite.schema.impl.ViewTable; > >>> +import org.apache.calcite.schema.impl.ViewTableMacro; > >>> +import org.apache.calcite.test.CalciteAssert; > >>> +import org.apache.calcite.test.ElasticChecker; > >>> + > >>> +import com.google.common.io.LineProcessor; > >>> +import com.google.common.io.Resources; > >>> + > >>> +import org.elasticsearch.action.bulk.BulkItemResponse; > >>> +import org.elasticsearch.action.bulk.BulkRequestBuilder; > >>> +import org.elasticsearch.action.bulk.BulkResponse; > >>> +import org.elasticsearch.common.xcontent.XContentBuilder; > >>> +import org.elasticsearch.common.xcontent.XContentFactory; > >>> + > >>> +import org.junit.BeforeClass; > >>> +import org.junit.ClassRule; > >>> +import org.junit.Test; > >>> + > >>> +import java.io.IOException; > >>> +import java.nio.charset.StandardCharsets; > >>> +import java.sql.Connection; > >>> +import java.sql.DriverManager; > >>> +import java.sql.SQLException; > >>> +import java.util.Arrays; > >>> +import java.util.Collections; > >>> +import java.util.Locale; > >>> + > >>> +/** > >>> + * Set of tests for ES adapter. Uses real instance via {@link > >> EmbeddedElasticRule}. Document > >>> + * source is local {@code zips-mini.json} file (located in the > >> classpath). > >>> + */ > >>> +public class ElasticSearch2AdapterTest { > >>> + > >>> + @ClassRule //init once for all tests > >>> + public static final EmbeddedElasticRule NODE = > >> EmbeddedElasticRule.create(); > >>> + > >>> + private static final String ZIPS = "zips"; > >>> + > >>> + /** > >>> + * Used to create {@code zips} index and insert some data > >>> + */ > >>> + @BeforeClass > >>> + public static void setupInstance() throws Exception { > >>> + // define mapping so fields are searchable (term query) > >>> + XContentBuilder mapping = > >> XContentFactory.jsonBuilder().startObject() > >>> + .startObject("properties") > >>> + .startObject("city").field("type", "string") > >>> + .field("index", "not_analyzed").endObject() > >>> + .startObject("state").field("type", "string") > >>> + .field("index", "not_analyzed").endObject() > >>> + .startObject("pop").field("type", "long").endObject() > >>> + .endObject() > >>> + .endObject(); > >>> + > >>> + // create index > >>> + NODE.client().admin().indices() > >>> + .prepareCreate(ZIPS) > >>> + .addMapping(ZIPS, mapping) > >>> + .get(); > >>> + > >>> + BulkRequestBuilder bulk = > >> NODE.client().prepareBulk().setRefresh(true); > >>> + > >>> + // load records from file > >>> + > >> > Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"), > >>> + StandardCharsets.UTF_8, new LineProcessor<Void>() { > >>> + @Override public boolean processLine(String line) throws > >> IOException { > >>> + line = line.replaceAll("_id", "id"); // _id is a > >> reserved attribute in ES > >>> + bulk.add(NODE.client().prepareIndex(ZIPS, > >> ZIPS).setSource(line)); > >>> + return true; > >>> + } > >>> + > >>> + @Override public Void getResult() { > >>> + return null; > >>> + } > >>> + }); > >>> + > >>> + if (bulk.numberOfActions() == 0) { > >>> + throw new IllegalStateException("No records to be indexed"); > >>> + } > >>> + > >>> + BulkResponse response = bulk.execute().get(); > >>> + > >>> + if (response.hasFailures()) { > >>> + throw new IllegalStateException( > >>> + String.format(Locale.getDefault(), "Failed to populate > >> %s:\n%s", NODE.httpAddress(), > >>> + > >> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed) > >>> + > >> > .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>"))); > >>> + } > >>> + > >>> + } > >>> + > >>> + private CalciteAssert.ConnectionFactory newConnectionFactory() { > >>> + return new CalciteAssert.ConnectionFactory() { > >>> + @Override public Connection createConnection() throws > >> SQLException { > >>> + final Connection connection = > >> DriverManager.getConnection("jdbc:calcite:"); > >>> + final SchemaPlus root = > >> connection.unwrap(CalciteConnection.class).getRootSchema(); > >>> + > >>> + root.add("elastic", new Elasticsearch2Schema(NODE.client(), > >> ZIPS)); > >>> + > >>> + // add calcite view programmatically > >>> + final String viewSql = "select cast(_MAP['city'] AS > >> varchar(20)) AS \"city\", " > >>> + + " cast(_MAP['loc'][0] AS float) AS \"longitude\",\n" > >>> + + " cast(_MAP['loc'][1] AS float) AS \"latitude\",\n" > >>> + + " cast(_MAP['pop'] AS integer) AS \"pop\", " > >>> + + " cast(_MAP['state'] AS varchar(2)) AS \"state\", " > >>> + + " cast(_MAP['id'] AS varchar(5)) AS \"id\" " > >>> + + "from \"elastic\".\"zips\""; > >>> + > >>> + ViewTableMacro macro = ViewTable.viewMacro(root, viewSql, > >>> + Collections.singletonList("elastic"), > >> Arrays.asList("elastic", "view"), false); > >>> + root.add("ZIPS", macro); > >>> + > >>> + return connection; > >>> + } > >>> + }; > >>> + } > >>> + > >>> + private CalciteAssert.AssertThat calciteAssert() { > >>> + return CalciteAssert.that() > >>> + .with(newConnectionFactory()); > >>> + } > >>> + > >>> + /** > >>> + * Tests using calcite view > >>> + */ > >>> + @Test > >>> + public void view() throws Exception { > >>> + calciteAssert() > >>> + .query("select * from zips where \"city\" = 'BROOKLYN'") > >>> + .returns("city=BROOKLYN; longitude=-73.956985; > >> latitude=40.646694; " > >>> + + "pop=111396; state=NY; id=11226\n") > >>> + .returnsCount(1); > >>> + } > >>> + > >>> + @Test > >>> + public void emptyResult() { > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from zips limit 0") > >>> + .returnsCount(0); > >>> + > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from \"elastic\".\"zips\" where > >> _MAP['Foo'] = '_MISSING_'") > >>> + .returnsCount(0); > >>> + } > >>> + > >>> + @Test > >>> + public void basic() throws Exception { > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from \"elastic\".\"zips\" where > >> _MAP['city'] = 'BROOKLYN'") > >>> + .returnsCount(1); > >>> + > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from \"elastic\".\"zips\" where" > >>> + + " _MAP['city'] in ('BROOKLYN', 'WASHINGTON')") > >>> + .returnsCount(2); > >>> + > >>> + // lower-case > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from \"elastic\".\"zips\" where " > >>> + + "_MAP['city'] in ('brooklyn', 'Brooklyn', > >> 'BROOK') ") > >>> + .returnsCount(0); > >>> + > >>> + // missing field > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from \"elastic\".\"zips\" where > >> _MAP['CITY'] = 'BROOKLYN'") > >>> + .returnsCount(0); > >>> + > >>> + // limit works > >>> + CalciteAssert.that() > >>> + .with(newConnectionFactory()) > >>> + .query("select * from \"elastic\".\"zips\" limit 42") > >>> + .returnsCount(42); > >>> + > >>> + } > >>> + > >>> + @Test public void testSort() { > >>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" > >>> + + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n" > >>> + + " ElasticsearchProject(city=[CAST(ITEM($0, > >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), > >> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], > >> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, > >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) > >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" > >>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; > >>> + > >>> + calciteAssert() > >>> + .query("select * from zips order by \"state\"") > >>> + .returnsCount(10) > >>> + .explainContains(explain); > >>> + } > >>> + > >>> + @Test public void testSortLimit() { > >>> + final String sql = "select \"state\", \"id\" from zips\n" > >>> + + "order by \"state\", \"id\" offset 2 rows fetch next 3 > >> rows only"; > >>> + calciteAssert() > >>> + .query(sql) > >>> + .returnsUnordered("state=AK; id=99801", > >>> + "state=AL; id=35215", > >>> + "state=AL; id=35401") > >>> + .queryContains( > >>> + ElasticChecker.elasticsearchChecker( > >>> + "\"_source\" : [\"state\", \"id\"]", > >>> + "\"sort\": [ {\"state\": \"asc\"}, > {\"id\": > >> \"asc\"}]", > >>> + "\"from\": 2", > >>> + "\"size\": 3")); > >>> + } > >>> + > >>> + > >>> + > >>> + @Test public void testOffsetLimit() { > >>> + final String sql = "select \"state\", \"id\" from zips\n" > >>> + + "offset 2 fetch next 3 rows only"; > >>> + calciteAssert() > >>> + .query(sql) > >>> + .runs() > >>> + .queryContains( > >>> + ElasticChecker.elasticsearchChecker( > >>> + "\"from\": 2", > >>> + "\"size\": 3", > >>> + "\"_source\" : [\"state\", \"id\"]")); > >>> + } > >>> + > >>> + @Test public void testLimit() { > >>> + final String sql = "select \"state\", \"id\" from zips\n" > >>> + + "fetch next 3 rows only"; > >>> + > >>> + calciteAssert() > >>> + .query(sql) > >>> + .runs() > >>> + .queryContains( > >>> + ElasticChecker.elasticsearchChecker( > >>> + "\"size\": 3", > >>> + "\"_source\" : [\"state\", \"id\"]")); > >>> + } > >>> + > >>> + @Test public void testFilterSort() { > >>> + final String sql = "select * from zips\n" > >>> + + "where \"state\" = 'CA' and \"id\" >= '70000'\n" > >>> + + "order by \"state\", \"id\""; > >>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" > >>> + + " ElasticsearchSort(sort0=[$4], sort1=[$5], dir0=[ASC], > >> dir1=[ASC])\n" > >>> + + " ElasticsearchProject(city=[CAST(ITEM($0, > >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), > >> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], > >> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, > >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) > >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" > >>> + + " > ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0, > >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0, 'id')):VARCHAR(5) > >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\", > >> '70000'))])\n" > >>> + + " ElasticsearchTableScan(table=[[elastic, > zips]])"; > >>> + calciteAssert() > >>> + .query(sql) > >>> + .returnsOrdered("city=LOS ANGELES; longitude=-118.258189; > >> latitude=34.007856; " > >>> + + "pop=96074; state=CA; id=90011", > >>> + "city=BELL GARDENS; longitude=-118.17205; > >> latitude=33.969177; " > >>> + + "pop=99568; state=CA; id=90201", > >>> + "city=NORWALK; longitude=-118.081767; > >> latitude=33.90564; " > >>> + + "pop=94188; state=CA; id=90650") > >>> + .queryContains( > >>> + ElasticChecker.elasticsearchChecker("\"query\" : " > >>> + + > >> "{\"constant_score\":{\"filter\":{\"bool\":" > >>> + + > >> "{\"must\":[{\"term\":{\"state\":\"CA\"}}," > >>> + + > >> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}", > >>> + "\"script_fields\": > >> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>> + + > >> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " > >>> + + "\"city\":{\"script\": > >> \"_source.city\"}, " > >>> + + "\"pop\":{\"script\": > >> \"_source.pop\"}, " > >>> + + "\"state\":{\"script\": > >> \"_source.state\"}, " > >>> + + "\"id\":{\"script\": \"_ > source.id > >> \"}}", > >>> + "\"sort\": [ {\"state\": \"asc\"}, > {\"id\": > >> \"asc\"}]")) > >>> + .explainContains(explain); > >>> + } > >>> + > >>> + @Test public void testFilterSortDesc() { > >>> + final String sql = "select * from zips\n" > >>> + + "where \"pop\" BETWEEN 95000 AND 100000\n" > >>> + + "order by \"state\" desc, \"pop\""; > >>> + calciteAssert() > >>> + .query(sql) > >>> + .limit(4) > >>> + .returnsOrdered( > >>> + "city=LOS ANGELES; longitude=-118.258189; > >> latitude=34.007856; pop=96074; state=CA; id=90011", > >>> + "city=BELL GARDENS; longitude=-118.17205; > >> latitude=33.969177; pop=99568; state=CA; id=90201"); > >>> + } > >>> + > >>> + @Test public void testFilterRedundant() { > >>> + final String sql = "select * from zips\n" > >>> + + "where \"state\" > 'CA' and \"state\" < 'AZ' and > >> \"state\" = 'OK'"; > >>> + calciteAssert() > >>> + .query(sql) > >>> + .runs() > >>> + .queryContains( > >>> + ElasticChecker.elasticsearchChecker("" > >>> + + "\"query\" : > >> {\"constant_score\":{\"filter\":{\"bool\":" > >>> + + > >> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}", > >>> + "\"script_fields\": > >> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>> + + > >> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " > >>> + + "\"city\":{\"script\": > >> \"_source.city\"}, " > >>> + + "\"pop\":{\"script\": > >> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, " > >>> + + "\"id\":{\"script\": > >> \"_source.id\"}}" > >>> + )); > >>> + } > >>> + > >>> + @Test public void testInPlan() { > >>> + final String[] searches = { > >>> + "\"query\" : > >> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":" > >>> + + > >> > "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":" > >>> + + "{\"pop\":99568}}]}}]}}}}", > >>> + "\"script_fields\": > >> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " > >>> + + "\"latitude\":{\"script\":\"_source.loc[1]\"}, " > >>> + + "\"city\":{\"script\": \"_source.city\"}, " > >>> + + "\"pop\":{\"script\": \"_source.pop\"}, " > >>> + + "\"state\":{\"script\": \"_source.state\"}, " > >>> + + "\"id\":{\"script\": \"_source.id\"}}" > >>> + }; > >>> + > >>> + calciteAssert() > >>> + .query("select * from zips where \"pop\" in (96074, > 99568)") > >>> + .returnsUnordered( > >>> + "city=BELL GARDENS; longitude=-118.17205; > >> latitude=33.969177; pop=99568; state=CA; id=90201", > >>> + "city=LOS ANGELES; longitude=-118.258189; > >> latitude=34.007856; pop=96074; state=CA; id=90011" > >>> + ) > >>> + > >> .queryContains(ElasticChecker.elasticsearchChecker(searches)); > >>> + } > >>> + > >>> + @Test public void testZips() { > >>> + calciteAssert() > >>> + .query("select \"state\", \"city\" from zips") > >>> + .returnsCount(10); > >>> + } > >>> + > >>> + @Test public void testProject() { > >>> + final String sql = "select \"state\", \"city\", 0 as \"zero\"\n" > >>> + + "from zips\n" > >>> + + "order by \"state\", \"city\""; > >>> + > >>> + calciteAssert() > >>> + .query(sql) > >>> + .limit(2) > >>> + .returnsUnordered("state=AK; city=ANCHORAGE; zero=0", > >>> + "state=AK; city=FAIRBANKS; zero=0") > >>> + .queryContains( > >>> + > >> ElasticChecker.elasticsearchChecker("\"script_fields\": " > >>> + + "{\"zero\":{\"script\": \"0\"}, > " > >>> + + "\"state\":{\"script\": > >> \"_source.state\"}, " > >>> + + "\"city\":{\"script\": > >> \"_source.city\"}}", > >>> + "\"sort\": [ {\"state\": \"asc\"}, > >> {\"city\": \"asc\"}]")); > >>> + } > >>> + > >>> + @Test public void testFilter() { > >>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" > >>> + + " ElasticsearchProject(state=[CAST(ITEM($0, > >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\"], city=[CAST(ITEM($0, 'city')):VARCHAR(20) > >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" > >>> + + " ElasticsearchFilter(condition=[=(CAST(ITEM($0, > >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE > >> \"ISO-8859-1$en_US$primary\", 'CA')])\n" > >>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; > >>> + calciteAssert() > >>> + .query("select \"state\", \"city\" from zips where > >> \"state\" = 'CA'") > >>> + .limit(3) > >>> + .returnsUnordered("state=CA; city=BELL GARDENS", > >>> + "state=CA; city=LOS ANGELES", > >>> + "state=CA; city=NORWALK") > >>> + .explainContains(explain); > >>> + } > >>> + > >>> + @Test public void testFilterReversed() { > >>> + calciteAssert() > >>> + .query("select \"state\", \"city\" from zips where 'WI' < > >> \"state\" order by \"city\"") > >>> + .limit(2) > >>> + .returnsUnordered("state=WV; city=BECKLEY", > >>> + "state=WY; city=CHEYENNE"); > >>> + calciteAssert() > >>> + .query("select \"state\", \"city\" from zips where \"state\" > >>> 'WI' order by \"city\"") > >>> + .limit(2) > >>> + .returnsUnordered("state=WV; city=BECKLEY", > >>> + "state=WY; city=CHEYENNE"); > >>> + } > >>> + > >>> +} > >>> + > >>> +// End ElasticSearch2AdapterTest.java > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>> new file mode 100644 > >>> index 0000000..4474add > >>> --- /dev/null > >>> +++ > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java > >>> @@ -0,0 +1,147 @@ > >>> +/* > >>> + * Licensed to the Apache Software Foundation (ASF) under one or more > >>> + * contributor license agreements. See the NOTICE file distributed > with > >>> + * this work for additional information regarding copyright ownership. > >>> + * The ASF licenses this file to you under the Apache License, Version > >> 2.0 > >>> + * (the "License"); you may not use this file except in compliance > with > >>> + * the License. You may obtain a copy of the License at > >>> + * > >>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>> + * > >>> + * Unless required by applicable law or agreed to in writing, software > >>> + * distributed under the License is distributed on an "AS IS" BASIS, > >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >> implied. > >>> + * See the License for the specific language governing permissions and > >>> + * limitations under the License. > >>> + */ > >>> +package org.apache.calcite.adapter.elasticsearch2; > >>> + > >>> +import com.google.common.base.Preconditions; > >>> +import com.google.common.io.Files; > >>> + > >>> +import org.elasticsearch.Version; > >>> +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo; > >>> +import org.elasticsearch.action.admin.cluster.node.info > >> .NodesInfoResponse; > >>> +import org.elasticsearch.client.Client; > >>> +import org.elasticsearch.common.settings.Settings; > >>> +import org.elasticsearch.common.transport.TransportAddress; > >>> +import org.elasticsearch.node.Node; > >>> +import org.elasticsearch.node.internal.InternalSettingsPreparer; > >>> +import org.elasticsearch.plugins.Plugin; > >>> +import org.elasticsearch.script.groovy.GroovyPlugin; > >>> + > >>> +import java.io.File; > >>> +import java.util.Arrays; > >>> +import java.util.Collection; > >>> +import java.util.Collections; > >>> + > >>> +/** > >>> + * Represents a single elastic search node which can run embedded in a > >> java application. > >>> + * Intended for unit and integration tests. Settings and plugins are > >> crafted for Calcite. > >>> + */ > >>> +class EmbeddedElasticNode implements AutoCloseable { > >>> + > >>> + private final LocalNode node; > >>> + private volatile boolean isStarted; > >>> + > >>> + private EmbeddedElasticNode(LocalNode node) { > >>> + this.node = Preconditions.checkNotNull(node, "node"); > >>> + } > >>> + > >>> + /** > >>> + * Having separate class to expose (protected) constructor which > >> allows to install > >>> + * different plugins. In our case it is {@code GroovyPlugin} for > >> scripted fields like > >>> + * {@code loc[0]} or {@code loc[1]['foo']}. > >>> + */ > >>> + private static class LocalNode extends Node { > >>> + private LocalNode(Settings settings, Collection<Class<? extends > >> Plugin>> classpathPlugins) { > >>> + super(InternalSettingsPreparer.prepareEnvironment(settings, > null), > >>> + Version.CURRENT, > >>> + classpathPlugins); > >>> + } > >>> + } > >>> + > >>> + /** > >>> + * Creates an instance with existing settings > >>> + */ > >>> + private static EmbeddedElasticNode create(Settings settings) { > >>> + // ensure GroovyPlugin is installed or otherwise scripted fields > >> would not work > >>> + LocalNode node = new LocalNode(settings, > >> Collections.singleton(GroovyPlugin.class)); > >>> + return new EmbeddedElasticNode(node); > >>> + } > >>> + > >>> + /** > >>> + * Creates elastic node as single member of a cluster. Node will not > >> be started > >>> + * unless {@link #start()} is explicitly called. > >>> + */ > >>> + public static EmbeddedElasticNode create() { > >>> + File data = Files.createTempDir(); > >>> + data.deleteOnExit(); > >>> + File home = Files.createTempDir(); > >>> + home.deleteOnExit(); > >>> + > >>> + Settings settings = Settings.builder() > >>> + .put("node.name", "fake-elastic") > >>> + .put("path.home", home.getAbsolutePath()) > >>> + .put("path.data", data.getAbsolutePath()) > >>> + .put("script.inline", true) // requires GroovyPlugin > >>> + .put("script.indexed", true) // requires GroovyPlugin > >>> + .put("cluster.routing.allocation.disk.threshold_enabled", > >> false) > >>> + .put("node.local", true) > >>> + .put("node.data", true) > >>> + .put("network.host", "localhost") > >>> + .build(); > >>> + > >>> + return create(settings); > >>> + } > >>> + > >>> + /** > >>> + * Starts current node > >>> + */ > >>> + public void start() { > >>> + Preconditions.checkState(!isStarted, "already started"); > >>> + node.start(); > >>> + this.isStarted = true; > >>> + } > >>> + > >>> + /** > >>> + * Returns current address to connect to with HTTP client. > >>> + */ > >>> + public TransportAddress httpAddress() { > >>> + Preconditions.checkState(isStarted, "node is not started"); > >>> + > >>> + NodesInfoResponse response = > >> client().admin().cluster().prepareNodesInfo() > >>> + .execute().actionGet(); > >>> + if (response.getNodes().length != 1) { > >>> + throw new IllegalStateException("Expected single node but got " > >>> + + response.getNodes().length); > >>> + } > >>> + NodeInfo node = response.getNodes()[0]; > >>> + return node.getHttp().address().boundAddresses()[0]; > >>> + } > >>> + > >>> + /** > >>> + * Exposes elastic > >>> + * <a href=" > >> > https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html > ">transport > >> client</a> > >>> + * > >>> + * (use of HTTP client is preferred). > >>> + */ > >>> + public Client client() { > >>> + Preconditions.checkState(isStarted, "node is not started"); > >>> + return node.client(); > >>> + } > >>> + > >>> + @Override public void close() throws Exception { > >>> + node.close(); > >>> + // cleanup data dirs > >>> + File data = new File(node.settings().get("path.data")); > >>> + File home = new File(node.settings().get("path.home")); > >>> + for (File file: Arrays.asList(data, home)) { > >>> + if (file.exists()) { > >>> + file.delete(); > >>> + } > >>> + } > >>> + } > >>> +} > >>> + > >>> +// End EmbeddedElasticNode.java > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java > >>> new file mode 100644 > >>> index 0000000..a633078 > >>> --- /dev/null > >>> +++ > >> > b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java > >>> @@ -0,0 +1,97 @@ > >>> +/* > >>> + * Licensed to the Apache Software Foundation (ASF) under one or more > >>> + * contributor license agreements. See the NOTICE file distributed > with > >>> + * this work for additional information regarding copyright ownership. > >>> + * The ASF licenses this file to you under the Apache License, Version > >> 2.0 > >>> + * (the "License"); you may not use this file except in compliance > with > >>> + * the License. You may obtain a copy of the License at > >>> + * > >>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>> + * > >>> + * Unless required by applicable law or agreed to in writing, software > >>> + * distributed under the License is distributed on an "AS IS" BASIS, > >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >> implied. > >>> + * See the License for the specific language governing permissions and > >>> + * limitations under the License. > >>> + */ > >>> +package org.apache.calcite.adapter.elasticsearch2; > >>> + > >>> +import com.google.common.base.Preconditions; > >>> + > >>> +import org.elasticsearch.client.Client; > >>> +import org.elasticsearch.common.transport.TransportAddress; > >>> +import org.junit.rules.ExternalResource; > >>> + > >>> +/** > >>> + * Used to initialize a single elastic node. For performance reasons > >> (node startup costs), > >>> + * same instance is usually shared across multiple tests. > >>> + * > >>> + * This rule should be used as follows: > >>> + * <pre> > >>> + * {@code > >>> + * > >>> + * public class MyTest { > >>> + * @literal @ClassRule > >>> + * public static final ElasticSearchRule RULE = > >> ElasticSearchRule.create(); > >>> + * > >>> + * @literal @BeforeClass > >>> + * public void setup() { > >>> + * // ... populate instance > >>> + * } > >>> + * > >>> + * @literal @Test > >>> + * public void myTest() { > >>> + * TransportAddress address = RULE.httpAddress(); > >>> + * // .... > >>> + * } > >>> + * } > >>> + * } > >>> + * </pre> > >>> + * > >>> + * @see ExternalResource > >>> + */ > >>> +class EmbeddedElasticRule extends ExternalResource { > >>> + > >>> + private final EmbeddedElasticNode node; > >>> + > >>> + private EmbeddedElasticRule(EmbeddedElasticNode resource) { > >>> + this.node = Preconditions.checkNotNull(resource, "resource"); > >>> + } > >>> + > >>> + @Override protected void before() throws Throwable { > >>> + node.start(); > >>> + } > >>> + > >>> + @Override protected void after() { > >>> + try { > >>> + node.close(); > >>> + } catch (Exception e) { > >>> + throw new RuntimeException(e); > >>> + } > >>> + } > >>> + > >>> + /** > >>> + * Factory method to create this rule. > >>> + */ > >>> + public static EmbeddedElasticRule create() { > >>> + return new EmbeddedElasticRule(EmbeddedElasticNode.create()); > >>> + } > >>> + > >>> + /** > >>> + * Exposes current ES transport client. > >>> + */ > >>> + Client client() { > >>> + return node.client(); > >>> + } > >>> + > >>> + /** > >>> + * HTTP address for rest clients (can be ES native or any other). > >>> + */ > >>> + TransportAddress httpAddress() { > >>> + return node.httpAddress(); > >>> + } > >>> + > >>> + > >>> +} > >>> + > >>> +// End EmbeddedElasticRule.java > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java > >> > b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java > >>> new file mode 100644 > >>> index 0000000..21fc491 > >>> --- /dev/null > >>> +++ > >> > b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java > >>> @@ -0,0 +1,49 @@ > >>> +/* > >>> + * Licensed to the Apache Software Foundation (ASF) under one or more > >>> + * contributor license agreements. See the NOTICE file distributed > with > >>> + * this work for additional information regarding copyright ownership. > >>> + * The ASF licenses this file to you under the Apache License, Version > >> 2.0 > >>> + * (the "License"); you may not use this file except in compliance > with > >>> + * the License. You may obtain a copy of the License at > >>> + * > >>> + * http://www.apache.org/licenses/LICENSE-2.0 > >>> + * > >>> + * Unless required by applicable law or agreed to in writing, software > >>> + * distributed under the License is distributed on an "AS IS" BASIS, > >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >> implied. > >>> + * See the License for the specific language governing permissions and > >>> + * limitations under the License. > >>> + */ > >>> +package org.apache.calcite.test; > >>> + > >>> +import com.google.common.base.Function; > >>> + > >>> +import java.util.List; > >>> + > >>> +import javax.annotation.Nullable; > >>> + > >>> +/** > >>> + * Internal util methods for ElasticSearch tests > >>> + */ > >>> +public class ElasticChecker { > >>> + > >>> + private ElasticChecker() {} > >>> + > >>> + > >>> + /** Returns a function that checks that a particular Elasticsearch > >> pipeline is > >>> + * generated to implement a query. */ > >>> + public static Function<List, Void> elasticsearchChecker(final > >> String... strings) { > >>> + return new Function<List, Void>() { > >>> + @Nullable > >>> + @Override public Void apply(@Nullable List actual) { > >>> + Object[] actualArray = actual == null || actual.isEmpty() ? > null > >>> + : ((List) actual.get(0)).toArray(); > >>> + CalciteAssert.assertArrayEqual("expected Elasticsearch query > >> not found", strings, > >>> + actualArray); > >>> + return null; > >>> + } > >>> + }; > >>> + } > >>> +} > >>> + > >>> +// End ElasticChecker.java > >>> > >>> > >> > http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java > >>> ---------------------------------------------------------------------- > >>> diff --git > >> > a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java > >> > b/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java > >>> deleted file mode 100644 > >>> index 4e0c2b6..0000000 > >>> --- > >> > a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java > >>> +++ /dev/null > >>> @@ -1,270 +0,0 @@ > >>> -/* > >>> - * Licensed to the Apache Software Foundation (ASF) under one or more > >>> - * contributor license agreements. See the NOTICE file distributed > with > >>> - * this work for additional information regarding copyright ownership. > >>> - * The ASF licenses this file to you under the Apache License, Version > >> 2.0 > >>> - * (the "License"); you may not use this file except in compliance > with > >>> - * the License. You may obtain a copy of the License at > >>> - * > >>> - * http://www.apache.org/licenses/LICENSE-2.0 > >>> - * > >>> - * Unless required by applicable law or agreed to in writing, software > >>> - * distributed under the License is distributed on an "AS IS" BASIS, > >>> - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > >> implied. > >>> - * See the License for the specific language governing permissions and > >>> - * limitations under the License. > >>> - */ > >>> -package org.apache.calcite.test; > >>> - > >>> -import org.apache.calcite.util.Util; > >>> - > >>> -import com.google.common.base.Function; > >>> -import com.google.common.collect.ImmutableMap; > >>> - > >>> -import org.junit.Test; > >>> - > >>> -import java.util.List; > >>> -import javax.annotation.Nullable; > >>> - > >>> -/** > >>> - * Tests for the {@code org.apache.calcite.adapter.elasticsearch2} > >> package. > >>> - * > >>> - * <p>Before calling this test, you need to populate Elasticsearch, as > >> follows: > >>> - * > >>> - * <blockquote><code> > >>> - * git clone https://github.com/vlsi/calcite-test-dataset<br> > >>> - * cd calcite-test-dataset<br> > >>> - * mvn install > >>> - * </code></blockquote> > >>> - * > >>> - * <p>This will create a virtual machine with Elastic
