I just tried it again. The following fails for me (jdk9, ubuntu): $ git checkout c12cb4b0de $ mvn clean -DskipTests site
Constructing Javadoc information... 1 error [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE [02:29 min] [INFO] Calcite Linq4j ..................................... SKIPPED [INFO] Calcite Core ....................................... SKIPPED [INFO] Calcite Cassandra .................................. SKIPPED [INFO] Calcite Druid ...................................... SKIPPED [INFO] Calcite Elasticsearch .............................. SKIPPED [INFO] Calcite Elasticsearch5 ............................. SKIPPED [INFO] Calcite Examples ................................... SKIPPED [INFO] Calcite Example CSV ................................ SKIPPED [INFO] Calcite Example Function ........................... SKIPPED [INFO] Calcite File ....................................... SKIPPED [INFO] Calcite Geode ...................................... SKIPPED [INFO] Calcite MongoDB .................................... SKIPPED [INFO] Calcite Pig ........................................ SKIPPED [INFO] Calcite Piglet ..................................... SKIPPED [INFO] Calcite Plus ....................................... SKIPPED [INFO] Calcite Server ..................................... SKIPPED [INFO] Calcite Spark ...................................... SKIPPED [INFO] Calcite Splunk ..................................... SKIPPED [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:29 min [INFO] Finished at: 2018-06-22T14:53:13-07:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate report: [ERROR] Exit code: 1 - /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: error: package org.elasticsearch.node.internal does not exist [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; [ERROR] ^ [ERROR] [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options @packages @argfile [ERROR] [ERROR] Refer to the generated Javadoc files in '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > On Jun 22, 2018, at 2:41 PM, Michael Mior <[email protected]> wrote: > > Odd, mvn clean site still works fine for me. > -- > Michael Mior > [email protected] > > > > Le ven. 22 juin 2018 à 17:12, Julian Hyde <[email protected]> a écrit : > >> Looks like this change broke “mvn site” (perhaps also “mvn >> javadoc:test-javadoc”). >> >> [ERROR] Failed to execute goal >> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on >> project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate >> report: >> [ERROR] Exit code: 1 - >> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: >> error: package org.elasticsearch.node.internal does not exist >> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer; >> [ERROR] ^ >> [ERROR] >> >>> On Jun 21, 2018, at 3:39 AM, [email protected] wrote: >>> >>> [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of >> ES adapter (Andrei Sereda) >>> >>> After discussion on dev-list Integration tests (for ES) have been >> removed. They're now >>> superseded by unit tests (which execute queries against a real elastic >> instance) >>> >>> Added local file (zips-mini.json) which contains a small subset of >> original zips.json >>> (allows to bootstrap tests faster) >>> >>> Created separate ES JUnit rule which can be re-used across different >> tests. >>> >>> Both v2 and v5 of ES adapters are supported. >>> >>> Close apache/calcite#716 >>> >>> >>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo >>> Commit: http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0 >>> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0 >>> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0 >>> >>> Branch: refs/heads/master >>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d >>> Parents: 37944bb >>> Author: Andrei Sereda <[email protected]> >>> Authored: Thu May 31 18:19:10 2018 -0400 >>> Committer: Michael Mior <[email protected]> >>> Committed: Thu Jun 21 06:38:50 2018 -0400 >>> >>> ---------------------------------------------------------------------- >>> .../AbstractElasticsearchTable.java | 12 + >>> .../elasticsearch/ElasticsearchProject.java | 61 ++- >>> elasticsearch2/pom.xml | 6 + >>> .../Elasticsearch2Enumerator.java | 12 +- >>> .../elasticsearch2/Elasticsearch2Schema.java | 16 +- >>> .../elasticsearch2/Elasticsearch2Table.java | 9 +- >>> .../ElasticSearch2AdapterTest.java | 395 ++++++++++++++++++ >>> .../elasticsearch2/EmbeddedElasticNode.java | 147 +++++++ >>> .../elasticsearch2/EmbeddedElasticRule.java | 97 +++++ >>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ >>> .../calcite/test/Elasticsearch2AdapterIT.java | 270 ------------- >>> .../resources/elasticsearch-zips-model.json | 50 --- >>> .../src/test/resources/zips-mini.json | 149 +++++++ >>> elasticsearch5/pom.xml | 31 ++ >>> .../elasticsearch5/Elasticsearch5Schema.java | 17 +- >>> .../elasticsearch5/Elasticsearch5Table.java | 11 +- >>> .../ElasticSearch5AdapterTest.java | 399 +++++++++++++++++++ >>> .../elasticsearch5/EmbeddedElasticNode.java | 153 +++++++ >>> .../elasticsearch5/EmbeddedElasticRule.java | 98 +++++ >>> .../org/apache/calcite/test/ElasticChecker.java | 49 +++ >>> .../calcite/test/Elasticsearch5AdapterIT.java | 270 ------------- >>> .../resources/elasticsearch-zips-model.json | 50 --- >>> elasticsearch5/src/test/resources/log4j2.xml | 16 + >>> .../src/test/resources/zips-mini.json | 149 +++++++ >>> pom.xml | 20 +- >>> 25 files changed, 1866 insertions(+), 670 deletions(-) >>> ---------------------------------------------------------------------- >>> >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>> index 0980469..8cc5933 100644 >>> --- >> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>> +++ >> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java >>> @@ -75,6 +75,18 @@ public abstract class AbstractElasticsearchTable >> extends AbstractQueryableTable >>> relOptTable, this, null); >>> } >>> >>> + /** >>> + * In ES 5.x scripted fields start with {@code params._source.foo} >> while in ES2.x >>> + * {@code _source.foo}. Helper method to build correct query based on >> runtime version of elastic. >>> + * >>> + * @see <a href=" >> https://github.com/elastic/elasticsearch/issues/20068";>_source >> variable</a> >>> + * @see <a href=" >> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html";>Scripted >> Fields</a> >>> + */ >>> + protected String scriptedFieldPrefix() { >>> + // this is default pattern starting 5.x >>> + return "params._source"; >>> + } >>> + >>> /** Executes a "find" operation on the underlying type. >>> * >>> * <p>For example, >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>> index b42abd7..961c8b0 100644 >>> --- >> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>> +++ >> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java >>> @@ -27,11 +27,15 @@ import >> org.apache.calcite.rel.metadata.RelMetadataQuery; >>> import org.apache.calcite.rel.type.RelDataType; >>> import org.apache.calcite.rex.RexNode; >>> import org.apache.calcite.util.Pair; >>> -import org.apache.calcite.util.Util; >>> + >>> +import com.google.common.base.Function; >>> +import com.google.common.collect.Lists; >>> >>> import java.util.ArrayList; >>> import java.util.List; >>> >>> +import javax.annotation.Nullable; >>> + >>> /** >>> * Implementation of {@link org.apache.calcite.rel.core.Project} >>> * relational expression in Elasticsearch. >>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends Project >> implements ElasticsearchRel { >>> implementor.visitChild(0, getInput()); >>> >>> final List<String> inFields = >>> - >> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); >>> + >> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType()); >>> final ElasticsearchRules.RexToElasticsearchTranslator translator = >>> - new ElasticsearchRules.RexToElasticsearchTranslator( >>> - (JavaTypeFactory) getCluster().getTypeFactory(), inFields); >>> + new ElasticsearchRules.RexToElasticsearchTranslator( >>> + (JavaTypeFactory) getCluster().getTypeFactory(), >> inFields); >>> >>> - final List<String> findItems = new ArrayList<>(); >>> - final List<String> scriptFieldItems = new ArrayList<>(); >>> + final List<String> fields = new ArrayList<>(); >>> + final List<String> scriptFields = new ArrayList<>(); >>> for (Pair<RexNode, String> pair: getNamedProjects()) { >>> final String name = pair.right; >>> final String expr = pair.left.accept(translator); >>> >>> if (expr.equals("\"" + name + "\"")) { >>> - findItems.add(ElasticsearchRules.quote(name)); >>> + fields.add(name); >>> } else if (expr.matches("\"literal\":.+")) { >>> - scriptFieldItems.add(ElasticsearchRules.quote(name) >>> - + ":{\"script\": " >>> - + expr.split(":")[1] + "}"); >>> + scriptFields.add(ElasticsearchRules.quote(name) >>> + + ":{\"script\": " >>> + + expr.split(":")[1] + "}"); >>> } else { >>> - scriptFieldItems.add(ElasticsearchRules.quote(name) >>> - + ":{\"script\":\"params._source." >>> - + expr.replaceAll("\"", "") + "\"}"); >>> + scriptFields.add(ElasticsearchRules.quote(name) >>> + + ":{\"script\":" >>> + // _source (ES2) vs params._source (ES5) >>> + + "\"" + >> implementor.elasticsearchTable.scriptedFieldPrefix() + "." >>> + + expr.replaceAll("\"", "") + "\"}"); >>> + } >>> + } >>> + >>> + StringBuilder query = new StringBuilder(); >>> + if (scriptFields.isEmpty()) { >>> + List<String> newList = Lists.transform(fields, new >> Function<String, String>() { >>> + @Nullable >>> + @Override public String apply(@Nullable String input) { >>> + return ElasticsearchRules.quote(input); >>> + } >>> + }); >>> + >>> + final String findString = String.join(", ", newList); >>> + query.append("\"_source\" : [").append(findString).append("]"); >>> + } else { >>> + // if scripted fields are present, ES ignores _source attribute >>> + for (String field: fields) { >>> + scriptFields.add(ElasticsearchRules.quote(field) + >> ":{\"script\": " >>> + // _source (ES2) vs params._source (ES5) >>> + + "\"" + >> implementor.elasticsearchTable.scriptedFieldPrefix() + "." >>> + + field + "\"}"); >>> } >>> + query.append("\"script_fields\": {" + String.join(", ", >> scriptFields) + "}"); >>> } >>> - final String findString = Util.toString(findItems, "", ", ", ""); >>> - final String scriptFieldString = "\"script_fields\": {" >>> - + Util.toString(scriptFieldItems, "", ", ", "") + "}"; >>> - final String fieldString = "\"_source\" : [" + findString + "]" >>> - + ", " + scriptFieldString; >>> >>> for (String opfield : implementor.list) { >>> if (opfield.startsWith("\"_source\"")) { >>> implementor.list.remove(opfield); >>> } >>> } >>> - implementor.add(fieldString); >>> + implementor.add(query.toString()); >>> } >>> } >>> >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml >>> ---------------------------------------------------------------------- >>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml >>> index f24622c..6fbee03 100644 >>> --- a/elasticsearch2/pom.xml >>> +++ b/elasticsearch2/pom.xml >>> @@ -73,6 +73,12 @@ limitations under the License. >>> <version>${elasticsearch-java-driver.version}</version> >>> </dependency> >>> <dependency> >>> + <!-- Lang groovy dependency is needed for testing with embedded >> ES (scripted fields like loc[0]) --> >>> + <groupId>org.elasticsearch.module</groupId> >>> + <artifactId>lang-groovy</artifactId> >>> + <scope>test</scope> >>> + </dependency> >>> + <dependency> >>> <groupId>com.carrotsearch</groupId> >>> <artifactId>hppc</artifactId> >>> <version>${hppc.version}</version> >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>> index 84370ab..c3d2ac0 100644 >>> --- >> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>> +++ >> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java >>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit; >>> import java.util.Date; >>> import java.util.Iterator; >>> import java.util.List; >>> +import java.util.Locale; >>> import java.util.Map; >>> >>> /** >>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator implements >> Enumerator<Object> { >>> private static Function1<SearchHit, Object[]> listGetter( >>> final List<Map.Entry<String, Class>> fields) { >>> return new Function1<SearchHit, Object[]>() { >>> - public Object[] apply(SearchHit searchHitFields) { >>> + public Object[] apply(SearchHit hit) { >>> Object[] objects = new Object[fields.size()]; >>> for (int i = 0; i < fields.size(); i++) { >>> final Map.Entry<String, Class> field = fields.get(i); >>> final String name = field.getKey(); >>> - if (searchHitFields.fields().isEmpty()) { >>> - objects[i] = convert(searchHitFields.getSource().get(name), >> field.getValue()); >>> + if (hit.fields().isEmpty()) { >>> + objects[i] = convert(hit.getSource().get(name), >> field.getValue()); >>> + } else if (hit.fields().containsKey(name)) { >>> + objects[i] = convert(hit.field(name).getValue(), >> field.getValue()); >>> } else { >>> - objects[i] = >> convert(searchHitFields.field(name).getValue(), field.getValue()); >>> + throw new IllegalStateException( >>> + String.format(Locale.getDefault(), "No result for >> %s", field)); >>> } >>> } >>> return objects; >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>> index 668402b..46e3fc5 100644 >>> --- >> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>> +++ >> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java >>> @@ -22,6 +22,9 @@ import org.apache.calcite.schema.impl.AbstractSchema; >>> >>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor; >>> >>> +import com.google.common.annotations.VisibleForTesting; >>> +import com.google.common.base.Preconditions; >>> + >>> import com.google.common.collect.ImmutableList; >>> import com.google.common.collect.ImmutableMap; >>> >>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends >> AbstractSchema >>> } >>> } >>> >>> + /** >>> + * Allows schema to be instantiated from existing elastic search >> client. >>> + * This constructor is used in tests. >>> + */ >>> + @VisibleForTesting >>> + Elasticsearch2Schema(Client client, String index) { >>> + this.client = Preconditions.checkNotNull(client, "client"); >>> + this.index = Preconditions.checkNotNull(index, "index"); >>> + } >>> + >>> @Override protected Map<String, Table> getTableMap() { >>> final ImmutableMap.Builder<String, Table> builder = >> ImmutableMap.builder(); >>> >>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends >> AbstractSchema >>> >>> final List<DiscoveryNode> nodes = >> ImmutableList.copyOf(transportClient.connectedNodes()); >>> if (nodes.isEmpty()) { >>> - throw new RuntimeException("Cannot connect to any elasticsearch >> nodes"); >>> + throw new IllegalStateException("Cannot connect to any >> elasticsearch node: " >>> + + transportNodes); >>> } >>> >>> client = transportClient; >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>> index 636aa5f..2928835 100644 >>> --- >> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>> +++ >> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java >>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends >> AbstractElasticsearchTable { >>> this.client = client; >>> } >>> >>> + /** >>> + * ES version 2.x. To access document attributes ES2 uses {@code >> _source.foo} syntax. >>> + */ >>> + @Override protected String scriptedFieldPrefix() { >>> + return "_source"; >>> + } >>> + >>> @Override protected Enumerable<Object> find(String index, List<String> >> ops, >>> - List<Map.Entry<String, Class>> fields) { >>> + List<Map.Entry<String, >> Class>> fields) { >>> final String dbName = index; >>> >>> final String queryString = "{" + Util.toString(ops, "", ", ", "") + >> "}"; >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>> new file mode 100644 >>> index 0000000..287e094 >>> --- /dev/null >>> +++ >> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java >>> @@ -0,0 +1,395 @@ >>> +/* >>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>> + * contributor license agreements. See the NOTICE file distributed with >>> + * this work for additional information regarding copyright ownership. >>> + * The ASF licenses this file to you under the Apache License, Version >> 2.0 >>> + * (the "License"); you may not use this file except in compliance with >>> + * the License. You may obtain a copy of the License at >>> + * >>> + * http://www.apache.org/licenses/LICENSE-2.0 >>> + * >>> + * Unless required by applicable law or agreed to in writing, software >>> + * distributed under the License is distributed on an "AS IS" BASIS, >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> implied. >>> + * See the License for the specific language governing permissions and >>> + * limitations under the License. >>> + */ >>> +package org.apache.calcite.adapter.elasticsearch2; >>> + >>> +import org.apache.calcite.jdbc.CalciteConnection; >>> +import org.apache.calcite.schema.SchemaPlus; >>> +import org.apache.calcite.schema.impl.ViewTable; >>> +import org.apache.calcite.schema.impl.ViewTableMacro; >>> +import org.apache.calcite.test.CalciteAssert; >>> +import org.apache.calcite.test.ElasticChecker; >>> + >>> +import com.google.common.io.LineProcessor; >>> +import com.google.common.io.Resources; >>> + >>> +import org.elasticsearch.action.bulk.BulkItemResponse; >>> +import org.elasticsearch.action.bulk.BulkRequestBuilder; >>> +import org.elasticsearch.action.bulk.BulkResponse; >>> +import org.elasticsearch.common.xcontent.XContentBuilder; >>> +import org.elasticsearch.common.xcontent.XContentFactory; >>> + >>> +import org.junit.BeforeClass; >>> +import org.junit.ClassRule; >>> +import org.junit.Test; >>> + >>> +import java.io.IOException; >>> +import java.nio.charset.StandardCharsets; >>> +import java.sql.Connection; >>> +import java.sql.DriverManager; >>> +import java.sql.SQLException; >>> +import java.util.Arrays; >>> +import java.util.Collections; >>> +import java.util.Locale; >>> + >>> +/** >>> + * Set of tests for ES adapter. Uses real instance via {@link >> EmbeddedElasticRule}. Document >>> + * source is local {@code zips-mini.json} file (located in the >> classpath). >>> + */ >>> +public class ElasticSearch2AdapterTest { >>> + >>> + @ClassRule //init once for all tests >>> + public static final EmbeddedElasticRule NODE = >> EmbeddedElasticRule.create(); >>> + >>> + private static final String ZIPS = "zips"; >>> + >>> + /** >>> + * Used to create {@code zips} index and insert some data >>> + */ >>> + @BeforeClass >>> + public static void setupInstance() throws Exception { >>> + // define mapping so fields are searchable (term query) >>> + XContentBuilder mapping = >> XContentFactory.jsonBuilder().startObject() >>> + .startObject("properties") >>> + .startObject("city").field("type", "string") >>> + .field("index", "not_analyzed").endObject() >>> + .startObject("state").field("type", "string") >>> + .field("index", "not_analyzed").endObject() >>> + .startObject("pop").field("type", "long").endObject() >>> + .endObject() >>> + .endObject(); >>> + >>> + // create index >>> + NODE.client().admin().indices() >>> + .prepareCreate(ZIPS) >>> + .addMapping(ZIPS, mapping) >>> + .get(); >>> + >>> + BulkRequestBuilder bulk = >> NODE.client().prepareBulk().setRefresh(true); >>> + >>> + // load records from file >>> + >> Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"), >>> + StandardCharsets.UTF_8, new LineProcessor<Void>() { >>> + @Override public boolean processLine(String line) throws >> IOException { >>> + line = line.replaceAll("_id", "id"); // _id is a >> reserved attribute in ES >>> + bulk.add(NODE.client().prepareIndex(ZIPS, >> ZIPS).setSource(line)); >>> + return true; >>> + } >>> + >>> + @Override public Void getResult() { >>> + return null; >>> + } >>> + }); >>> + >>> + if (bulk.numberOfActions() == 0) { >>> + throw new IllegalStateException("No records to be indexed"); >>> + } >>> + >>> + BulkResponse response = bulk.execute().get(); >>> + >>> + if (response.hasFailures()) { >>> + throw new IllegalStateException( >>> + String.format(Locale.getDefault(), "Failed to populate >> %s:\n%s", NODE.httpAddress(), >>> + >> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed) >>> + >> .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>"))); >>> + } >>> + >>> + } >>> + >>> + private CalciteAssert.ConnectionFactory newConnectionFactory() { >>> + return new CalciteAssert.ConnectionFactory() { >>> + @Override public Connection createConnection() throws >> SQLException { >>> + final Connection connection = >> DriverManager.getConnection("jdbc:calcite:"); >>> + final SchemaPlus root = >> connection.unwrap(CalciteConnection.class).getRootSchema(); >>> + >>> + root.add("elastic", new Elasticsearch2Schema(NODE.client(), >> ZIPS)); >>> + >>> + // add calcite view programmatically >>> + final String viewSql = "select cast(_MAP['city'] AS >> varchar(20)) AS \"city\", " >>> + + " cast(_MAP['loc'][0] AS float) AS \"longitude\",\n" >>> + + " cast(_MAP['loc'][1] AS float) AS \"latitude\",\n" >>> + + " cast(_MAP['pop'] AS integer) AS \"pop\", " >>> + + " cast(_MAP['state'] AS varchar(2)) AS \"state\", " >>> + + " cast(_MAP['id'] AS varchar(5)) AS \"id\" " >>> + + "from \"elastic\".\"zips\""; >>> + >>> + ViewTableMacro macro = ViewTable.viewMacro(root, viewSql, >>> + Collections.singletonList("elastic"), >> Arrays.asList("elastic", "view"), false); >>> + root.add("ZIPS", macro); >>> + >>> + return connection; >>> + } >>> + }; >>> + } >>> + >>> + private CalciteAssert.AssertThat calciteAssert() { >>> + return CalciteAssert.that() >>> + .with(newConnectionFactory()); >>> + } >>> + >>> + /** >>> + * Tests using calcite view >>> + */ >>> + @Test >>> + public void view() throws Exception { >>> + calciteAssert() >>> + .query("select * from zips where \"city\" = 'BROOKLYN'") >>> + .returns("city=BROOKLYN; longitude=-73.956985; >> latitude=40.646694; " >>> + + "pop=111396; state=NY; id=11226\n") >>> + .returnsCount(1); >>> + } >>> + >>> + @Test >>> + public void emptyResult() { >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from zips limit 0") >>> + .returnsCount(0); >>> + >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from \"elastic\".\"zips\" where >> _MAP['Foo'] = '_MISSING_'") >>> + .returnsCount(0); >>> + } >>> + >>> + @Test >>> + public void basic() throws Exception { >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from \"elastic\".\"zips\" where >> _MAP['city'] = 'BROOKLYN'") >>> + .returnsCount(1); >>> + >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from \"elastic\".\"zips\" where" >>> + + " _MAP['city'] in ('BROOKLYN', 'WASHINGTON')") >>> + .returnsCount(2); >>> + >>> + // lower-case >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from \"elastic\".\"zips\" where " >>> + + "_MAP['city'] in ('brooklyn', 'Brooklyn', >> 'BROOK') ") >>> + .returnsCount(0); >>> + >>> + // missing field >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from \"elastic\".\"zips\" where >> _MAP['CITY'] = 'BROOKLYN'") >>> + .returnsCount(0); >>> + >>> + // limit works >>> + CalciteAssert.that() >>> + .with(newConnectionFactory()) >>> + .query("select * from \"elastic\".\"zips\" limit 42") >>> + .returnsCount(42); >>> + >>> + } >>> + >>> + @Test public void testSort() { >>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>> + + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n" >>> + + " ElasticsearchProject(city=[CAST(ITEM($0, >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), >> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], >> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; >>> + >>> + calciteAssert() >>> + .query("select * from zips order by \"state\"") >>> + .returnsCount(10) >>> + .explainContains(explain); >>> + } >>> + >>> + @Test public void testSortLimit() { >>> + final String sql = "select \"state\", \"id\" from zips\n" >>> + + "order by \"state\", \"id\" offset 2 rows fetch next 3 >> rows only"; >>> + calciteAssert() >>> + .query(sql) >>> + .returnsUnordered("state=AK; id=99801", >>> + "state=AL; id=35215", >>> + "state=AL; id=35401") >>> + .queryContains( >>> + ElasticChecker.elasticsearchChecker( >>> + "\"_source\" : [\"state\", \"id\"]", >>> + "\"sort\": [ {\"state\": \"asc\"}, {\"id\": >> \"asc\"}]", >>> + "\"from\": 2", >>> + "\"size\": 3")); >>> + } >>> + >>> + >>> + >>> + @Test public void testOffsetLimit() { >>> + final String sql = "select \"state\", \"id\" from zips\n" >>> + + "offset 2 fetch next 3 rows only"; >>> + calciteAssert() >>> + .query(sql) >>> + .runs() >>> + .queryContains( >>> + ElasticChecker.elasticsearchChecker( >>> + "\"from\": 2", >>> + "\"size\": 3", >>> + "\"_source\" : [\"state\", \"id\"]")); >>> + } >>> + >>> + @Test public void testLimit() { >>> + final String sql = "select \"state\", \"id\" from zips\n" >>> + + "fetch next 3 rows only"; >>> + >>> + calciteAssert() >>> + .query(sql) >>> + .runs() >>> + .queryContains( >>> + ElasticChecker.elasticsearchChecker( >>> + "\"size\": 3", >>> + "\"_source\" : [\"state\", \"id\"]")); >>> + } >>> + >>> + @Test public void testFilterSort() { >>> + final String sql = "select * from zips\n" >>> + + "where \"state\" = 'CA' and \"id\" >= '70000'\n" >>> + + "order by \"state\", \"id\""; >>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>> + + " ElasticsearchSort(sort0=[$4], sort1=[$5], dir0=[ASC], >> dir1=[ASC])\n" >>> + + " ElasticsearchProject(city=[CAST(ITEM($0, >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), >> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], >> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>> + + " ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0, 'id')):VARCHAR(5) >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\", >> '70000'))])\n" >>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; >>> + calciteAssert() >>> + .query(sql) >>> + .returnsOrdered("city=LOS ANGELES; longitude=-118.258189; >> latitude=34.007856; " >>> + + "pop=96074; state=CA; id=90011", >>> + "city=BELL GARDENS; longitude=-118.17205; >> latitude=33.969177; " >>> + + "pop=99568; state=CA; id=90201", >>> + "city=NORWALK; longitude=-118.081767; >> latitude=33.90564; " >>> + + "pop=94188; state=CA; id=90650") >>> + .queryContains( >>> + ElasticChecker.elasticsearchChecker("\"query\" : " >>> + + >> "{\"constant_score\":{\"filter\":{\"bool\":" >>> + + >> "{\"must\":[{\"term\":{\"state\":\"CA\"}}," >>> + + >> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}", >>> + "\"script_fields\": >> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " >>> + + >> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " >>> + + "\"city\":{\"script\": >> \"_source.city\"}, " >>> + + "\"pop\":{\"script\": >> \"_source.pop\"}, " >>> + + "\"state\":{\"script\": >> \"_source.state\"}, " >>> + + "\"id\":{\"script\": \"_source.id >> \"}}", >>> + "\"sort\": [ {\"state\": \"asc\"}, {\"id\": >> \"asc\"}]")) >>> + .explainContains(explain); >>> + } >>> + >>> + @Test public void testFilterSortDesc() { >>> + final String sql = "select * from zips\n" >>> + + "where \"pop\" BETWEEN 95000 AND 100000\n" >>> + + "order by \"state\" desc, \"pop\""; >>> + calciteAssert() >>> + .query(sql) >>> + .limit(4) >>> + .returnsOrdered( >>> + "city=LOS ANGELES; longitude=-118.258189; >> latitude=34.007856; pop=96074; state=CA; id=90011", >>> + "city=BELL GARDENS; longitude=-118.17205; >> latitude=33.969177; pop=99568; state=CA; id=90201"); >>> + } >>> + >>> + @Test public void testFilterRedundant() { >>> + final String sql = "select * from zips\n" >>> + + "where \"state\" > 'CA' and \"state\" < 'AZ' and >> \"state\" = 'OK'"; >>> + calciteAssert() >>> + .query(sql) >>> + .runs() >>> + .queryContains( >>> + ElasticChecker.elasticsearchChecker("" >>> + + "\"query\" : >> {\"constant_score\":{\"filter\":{\"bool\":" >>> + + >> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}", >>> + "\"script_fields\": >> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " >>> + + >> "\"latitude\":{\"script\":\"_source.loc[1]\"}, " >>> + + "\"city\":{\"script\": >> \"_source.city\"}, " >>> + + "\"pop\":{\"script\": >> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, " >>> + + "\"id\":{\"script\": >> \"_source.id\"}}" >>> + )); >>> + } >>> + >>> + @Test public void testInPlan() { >>> + final String[] searches = { >>> + "\"query\" : >> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":" >>> + + >> "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":" >>> + + "{\"pop\":99568}}]}}]}}}}", >>> + "\"script_fields\": >> {\"longitude\":{\"script\":\"_source.loc[0]\"}, " >>> + + "\"latitude\":{\"script\":\"_source.loc[1]\"}, " >>> + + "\"city\":{\"script\": \"_source.city\"}, " >>> + + "\"pop\":{\"script\": \"_source.pop\"}, " >>> + + "\"state\":{\"script\": \"_source.state\"}, " >>> + + "\"id\":{\"script\": \"_source.id\"}}" >>> + }; >>> + >>> + calciteAssert() >>> + .query("select * from zips where \"pop\" in (96074, 99568)") >>> + .returnsUnordered( >>> + "city=BELL GARDENS; longitude=-118.17205; >> latitude=33.969177; pop=99568; state=CA; id=90201", >>> + "city=LOS ANGELES; longitude=-118.258189; >> latitude=34.007856; pop=96074; state=CA; id=90011" >>> + ) >>> + >> .queryContains(ElasticChecker.elasticsearchChecker(searches)); >>> + } >>> + >>> + @Test public void testZips() { >>> + calciteAssert() >>> + .query("select \"state\", \"city\" from zips") >>> + .returnsCount(10); >>> + } >>> + >>> + @Test public void testProject() { >>> + final String sql = "select \"state\", \"city\", 0 as \"zero\"\n" >>> + + "from zips\n" >>> + + "order by \"state\", \"city\""; >>> + >>> + calciteAssert() >>> + .query(sql) >>> + .limit(2) >>> + .returnsUnordered("state=AK; city=ANCHORAGE; zero=0", >>> + "state=AK; city=FAIRBANKS; zero=0") >>> + .queryContains( >>> + >> ElasticChecker.elasticsearchChecker("\"script_fields\": " >>> + + "{\"zero\":{\"script\": \"0\"}, " >>> + + "\"state\":{\"script\": >> \"_source.state\"}, " >>> + + "\"city\":{\"script\": >> \"_source.city\"}}", >>> + "\"sort\": [ {\"state\": \"asc\"}, >> {\"city\": \"asc\"}]")); >>> + } >>> + >>> + @Test public void testFilter() { >>> + final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>> + + " ElasticsearchProject(state=[CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], city=[CAST(ITEM($0, 'city')):VARCHAR(20) >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>> + + " ElasticsearchFilter(condition=[=(CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\", 'CA')])\n" >>> + + " ElasticsearchTableScan(table=[[elastic, zips]])"; >>> + calciteAssert() >>> + .query("select \"state\", \"city\" from zips where >> \"state\" = 'CA'") >>> + .limit(3) >>> + .returnsUnordered("state=CA; city=BELL GARDENS", >>> + "state=CA; city=LOS ANGELES", >>> + "state=CA; city=NORWALK") >>> + .explainContains(explain); >>> + } >>> + >>> + @Test public void testFilterReversed() { >>> + calciteAssert() >>> + .query("select \"state\", \"city\" from zips where 'WI' < >> \"state\" order by \"city\"") >>> + .limit(2) >>> + .returnsUnordered("state=WV; city=BECKLEY", >>> + "state=WY; city=CHEYENNE"); >>> + calciteAssert() >>> + .query("select \"state\", \"city\" from zips where \"state\" >>> 'WI' order by \"city\"") >>> + .limit(2) >>> + .returnsUnordered("state=WV; city=BECKLEY", >>> + "state=WY; city=CHEYENNE"); >>> + } >>> + >>> +} >>> + >>> +// End ElasticSearch2AdapterTest.java >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>> new file mode 100644 >>> index 0000000..4474add >>> --- /dev/null >>> +++ >> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java >>> @@ -0,0 +1,147 @@ >>> +/* >>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>> + * contributor license agreements. See the NOTICE file distributed with >>> + * this work for additional information regarding copyright ownership. >>> + * The ASF licenses this file to you under the Apache License, Version >> 2.0 >>> + * (the "License"); you may not use this file except in compliance with >>> + * the License. You may obtain a copy of the License at >>> + * >>> + * http://www.apache.org/licenses/LICENSE-2.0 >>> + * >>> + * Unless required by applicable law or agreed to in writing, software >>> + * distributed under the License is distributed on an "AS IS" BASIS, >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> implied. >>> + * See the License for the specific language governing permissions and >>> + * limitations under the License. >>> + */ >>> +package org.apache.calcite.adapter.elasticsearch2; >>> + >>> +import com.google.common.base.Preconditions; >>> +import com.google.common.io.Files; >>> + >>> +import org.elasticsearch.Version; >>> +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo; >>> +import org.elasticsearch.action.admin.cluster.node.info >> .NodesInfoResponse; >>> +import org.elasticsearch.client.Client; >>> +import org.elasticsearch.common.settings.Settings; >>> +import org.elasticsearch.common.transport.TransportAddress; >>> +import org.elasticsearch.node.Node; >>> +import org.elasticsearch.node.internal.InternalSettingsPreparer; >>> +import org.elasticsearch.plugins.Plugin; >>> +import org.elasticsearch.script.groovy.GroovyPlugin; >>> + >>> +import java.io.File; >>> +import java.util.Arrays; >>> +import java.util.Collection; >>> +import java.util.Collections; >>> + >>> +/** >>> + * Represents a single elastic search node which can run embedded in a >> java application. >>> + * Intended for unit and integration tests. Settings and plugins are >> crafted for Calcite. >>> + */ >>> +class EmbeddedElasticNode implements AutoCloseable { >>> + >>> + private final LocalNode node; >>> + private volatile boolean isStarted; >>> + >>> + private EmbeddedElasticNode(LocalNode node) { >>> + this.node = Preconditions.checkNotNull(node, "node"); >>> + } >>> + >>> + /** >>> + * Having separate class to expose (protected) constructor which >> allows to install >>> + * different plugins. In our case it is {@code GroovyPlugin} for >> scripted fields like >>> + * {@code loc[0]} or {@code loc[1]['foo']}. >>> + */ >>> + private static class LocalNode extends Node { >>> + private LocalNode(Settings settings, Collection<Class<? extends >> Plugin>> classpathPlugins) { >>> + super(InternalSettingsPreparer.prepareEnvironment(settings, null), >>> + Version.CURRENT, >>> + classpathPlugins); >>> + } >>> + } >>> + >>> + /** >>> + * Creates an instance with existing settings >>> + */ >>> + private static EmbeddedElasticNode create(Settings settings) { >>> + // ensure GroovyPlugin is installed or otherwise scripted fields >> would not work >>> + LocalNode node = new LocalNode(settings, >> Collections.singleton(GroovyPlugin.class)); >>> + return new EmbeddedElasticNode(node); >>> + } >>> + >>> + /** >>> + * Creates elastic node as single member of a cluster. Node will not >> be started >>> + * unless {@link #start()} is explicitly called. >>> + */ >>> + public static EmbeddedElasticNode create() { >>> + File data = Files.createTempDir(); >>> + data.deleteOnExit(); >>> + File home = Files.createTempDir(); >>> + home.deleteOnExit(); >>> + >>> + Settings settings = Settings.builder() >>> + .put("node.name", "fake-elastic") >>> + .put("path.home", home.getAbsolutePath()) >>> + .put("path.data", data.getAbsolutePath()) >>> + .put("script.inline", true) // requires GroovyPlugin >>> + .put("script.indexed", true) // requires GroovyPlugin >>> + .put("cluster.routing.allocation.disk.threshold_enabled", >> false) >>> + .put("node.local", true) >>> + .put("node.data", true) >>> + .put("network.host", "localhost") >>> + .build(); >>> + >>> + return create(settings); >>> + } >>> + >>> + /** >>> + * Starts current node >>> + */ >>> + public void start() { >>> + Preconditions.checkState(!isStarted, "already started"); >>> + node.start(); >>> + this.isStarted = true; >>> + } >>> + >>> + /** >>> + * Returns current address to connect to with HTTP client. >>> + */ >>> + public TransportAddress httpAddress() { >>> + Preconditions.checkState(isStarted, "node is not started"); >>> + >>> + NodesInfoResponse response = >> client().admin().cluster().prepareNodesInfo() >>> + .execute().actionGet(); >>> + if (response.getNodes().length != 1) { >>> + throw new IllegalStateException("Expected single node but got " >>> + + response.getNodes().length); >>> + } >>> + NodeInfo node = response.getNodes()[0]; >>> + return node.getHttp().address().boundAddresses()[0]; >>> + } >>> + >>> + /** >>> + * Exposes elastic >>> + * <a href=" >> https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html";>transport >> client</a> >>> + * >>> + * (use of HTTP client is preferred). >>> + */ >>> + public Client client() { >>> + Preconditions.checkState(isStarted, "node is not started"); >>> + return node.client(); >>> + } >>> + >>> + @Override public void close() throws Exception { >>> + node.close(); >>> + // cleanup data dirs >>> + File data = new File(node.settings().get("path.data")); >>> + File home = new File(node.settings().get("path.home")); >>> + for (File file: Arrays.asList(data, home)) { >>> + if (file.exists()) { >>> + file.delete(); >>> + } >>> + } >>> + } >>> +} >>> + >>> +// End EmbeddedElasticNode.java >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>> new file mode 100644 >>> index 0000000..a633078 >>> --- /dev/null >>> +++ >> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticRule.java >>> @@ -0,0 +1,97 @@ >>> +/* >>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>> + * contributor license agreements. See the NOTICE file distributed with >>> + * this work for additional information regarding copyright ownership. >>> + * The ASF licenses this file to you under the Apache License, Version >> 2.0 >>> + * (the "License"); you may not use this file except in compliance with >>> + * the License. You may obtain a copy of the License at >>> + * >>> + * http://www.apache.org/licenses/LICENSE-2.0 >>> + * >>> + * Unless required by applicable law or agreed to in writing, software >>> + * distributed under the License is distributed on an "AS IS" BASIS, >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> implied. >>> + * See the License for the specific language governing permissions and >>> + * limitations under the License. >>> + */ >>> +package org.apache.calcite.adapter.elasticsearch2; >>> + >>> +import com.google.common.base.Preconditions; >>> + >>> +import org.elasticsearch.client.Client; >>> +import org.elasticsearch.common.transport.TransportAddress; >>> +import org.junit.rules.ExternalResource; >>> + >>> +/** >>> + * Used to initialize a single elastic node. For performance reasons >> (node startup costs), >>> + * same instance is usually shared across multiple tests. >>> + * >>> + * This rule should be used as follows: >>> + * <pre> >>> + * {@code >>> + * >>> + * public class MyTest { >>> + * @literal @ClassRule >>> + * public static final ElasticSearchRule RULE = >> ElasticSearchRule.create(); >>> + * >>> + * @literal @BeforeClass >>> + * public void setup() { >>> + * // ... populate instance >>> + * } >>> + * >>> + * @literal @Test >>> + * public void myTest() { >>> + * TransportAddress address = RULE.httpAddress(); >>> + * // .... >>> + * } >>> + * } >>> + * } >>> + * </pre> >>> + * >>> + * @see ExternalResource >>> + */ >>> +class EmbeddedElasticRule extends ExternalResource { >>> + >>> + private final EmbeddedElasticNode node; >>> + >>> + private EmbeddedElasticRule(EmbeddedElasticNode resource) { >>> + this.node = Preconditions.checkNotNull(resource, "resource"); >>> + } >>> + >>> + @Override protected void before() throws Throwable { >>> + node.start(); >>> + } >>> + >>> + @Override protected void after() { >>> + try { >>> + node.close(); >>> + } catch (Exception e) { >>> + throw new RuntimeException(e); >>> + } >>> + } >>> + >>> + /** >>> + * Factory method to create this rule. >>> + */ >>> + public static EmbeddedElasticRule create() { >>> + return new EmbeddedElasticRule(EmbeddedElasticNode.create()); >>> + } >>> + >>> + /** >>> + * Exposes current ES transport client. >>> + */ >>> + Client client() { >>> + return node.client(); >>> + } >>> + >>> + /** >>> + * HTTP address for rest clients (can be ES native or any other). >>> + */ >>> + TransportAddress httpAddress() { >>> + return node.httpAddress(); >>> + } >>> + >>> + >>> +} >>> + >>> +// End EmbeddedElasticRule.java >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >> b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>> new file mode 100644 >>> index 0000000..21fc491 >>> --- /dev/null >>> +++ >> b/elasticsearch2/src/test/java/org/apache/calcite/test/ElasticChecker.java >>> @@ -0,0 +1,49 @@ >>> +/* >>> + * Licensed to the Apache Software Foundation (ASF) under one or more >>> + * contributor license agreements. See the NOTICE file distributed with >>> + * this work for additional information regarding copyright ownership. >>> + * The ASF licenses this file to you under the Apache License, Version >> 2.0 >>> + * (the "License"); you may not use this file except in compliance with >>> + * the License. You may obtain a copy of the License at >>> + * >>> + * http://www.apache.org/licenses/LICENSE-2.0 >>> + * >>> + * Unless required by applicable law or agreed to in writing, software >>> + * distributed under the License is distributed on an "AS IS" BASIS, >>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> implied. >>> + * See the License for the specific language governing permissions and >>> + * limitations under the License. >>> + */ >>> +package org.apache.calcite.test; >>> + >>> +import com.google.common.base.Function; >>> + >>> +import java.util.List; >>> + >>> +import javax.annotation.Nullable; >>> + >>> +/** >>> + * Internal util methods for ElasticSearch tests >>> + */ >>> +public class ElasticChecker { >>> + >>> + private ElasticChecker() {} >>> + >>> + >>> + /** Returns a function that checks that a particular Elasticsearch >> pipeline is >>> + * generated to implement a query. */ >>> + public static Function<List, Void> elasticsearchChecker(final >> String... strings) { >>> + return new Function<List, Void>() { >>> + @Nullable >>> + @Override public Void apply(@Nullable List actual) { >>> + Object[] actualArray = actual == null || actual.isEmpty() ? null >>> + : ((List) actual.get(0)).toArray(); >>> + CalciteAssert.assertArrayEqual("expected Elasticsearch query >> not found", strings, >>> + actualArray); >>> + return null; >>> + } >>> + }; >>> + } >>> +} >>> + >>> +// End ElasticChecker.java >>> >>> >> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>> ---------------------------------------------------------------------- >>> diff --git >> a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >> b/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>> deleted file mode 100644 >>> index 4e0c2b6..0000000 >>> --- >> a/elasticsearch2/src/test/java/org/apache/calcite/test/Elasticsearch2AdapterIT.java >>> +++ /dev/null >>> @@ -1,270 +0,0 @@ >>> -/* >>> - * Licensed to the Apache Software Foundation (ASF) under one or more >>> - * contributor license agreements. See the NOTICE file distributed with >>> - * this work for additional information regarding copyright ownership. >>> - * The ASF licenses this file to you under the Apache License, Version >> 2.0 >>> - * (the "License"); you may not use this file except in compliance with >>> - * the License. You may obtain a copy of the License at >>> - * >>> - * http://www.apache.org/licenses/LICENSE-2.0 >>> - * >>> - * Unless required by applicable law or agreed to in writing, software >>> - * distributed under the License is distributed on an "AS IS" BASIS, >>> - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> implied. >>> - * See the License for the specific language governing permissions and >>> - * limitations under the License. >>> - */ >>> -package org.apache.calcite.test; >>> - >>> -import org.apache.calcite.util.Util; >>> - >>> -import com.google.common.base.Function; >>> -import com.google.common.collect.ImmutableMap; >>> - >>> -import org.junit.Test; >>> - >>> -import java.util.List; >>> -import javax.annotation.Nullable; >>> - >>> -/** >>> - * Tests for the {@code org.apache.calcite.adapter.elasticsearch2} >> package. >>> - * >>> - * <p>Before calling this test, you need to populate Elasticsearch, as >> follows: >>> - * >>> - * <blockquote><code> >>> - * git clone https://github.com/vlsi/calcite-test-dataset<br> >>> - * cd calcite-test-dataset<br> >>> - * mvn install >>> - * </code></blockquote> >>> - * >>> - * <p>This will create a virtual machine with Elasticsearch and the >> "zips" test >>> - * dataset. >>> - */ >>> -public class Elasticsearch2AdapterIT { >>> - /** >>> - * Whether to run Elasticsearch tests. Enabled by default, however >> test is only >>> - * included if "it" profile is activated ({@code -Pit}). To disable, >>> - * specify {@code -Dcalcite.test.elasticsearch=false} on the Java >> command line. >>> - */ >>> - private static final boolean ENABLED = >> Util.getBooleanProperty("calcite.test.elasticsearch", >>> - true); >>> - >>> - /** Connection factory based on the "zips-es" model. */ >>> - private static final ImmutableMap<String, String> ZIPS = >> ImmutableMap.of("model", >>> - >> Elasticsearch2AdapterIT.class.getResource("/elasticsearch-zips-model.json").getPath()); >>> - >>> - /** Whether to run this test. */ >>> - private boolean enabled() { >>> - return ENABLED; >>> - } >>> - >>> - /** Returns a function that checks that a particular Elasticsearch >> pipeline is >>> - * generated to implement a query. */ >>> - private static Function<List, Void> elasticsearchChecker(final >> String... strings) { >>> - return new Function<List, Void>() { >>> - @Nullable >>> - @Override public Void apply(@Nullable List actual) { >>> - Object[] actualArray = actual == null || actual.isEmpty() ? null >>> - : ((List) actual.get(0)).toArray(); >>> - CalciteAssert.assertArrayEqual("expected Elasticsearch query >> not found", strings, >>> - actualArray); >>> - return null; >>> - } >>> - }; >>> - } >>> - >>> - @Test public void testSort() { >>> - final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>> - + " ElasticsearchSort(sort0=[$4], dir0=[ASC])\n" >>> - + " ElasticsearchProject(city=[CAST(ITEM($0, >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), >> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], >> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>> - + " ElasticsearchTableScan(table=[[elasticsearch_raw, >> zips]])"; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query("select * from zips order by \"state\"") >>> - .returnsCount(10) >>> - .explainContains(explain); >>> - } >>> - >>> - @Test public void testSortLimit() { >>> - final String sql = "select \"state\", \"id\" from zips\n" >>> - + "order by \"state\", \"id\" offset 2 rows fetch next 3 rows >> only"; >>> - CalciteAssert.that() >>> - .with(ZIPS) >>> - .query(sql) >>> - .returnsUnordered("state=AK; id=99503", >>> - "state=AK; id=99504", >>> - "state=AK; id=99505") >>> - .queryContains( >>> - elasticsearchChecker( >>> - "\"fields\" : [\"state\", \"id\"], \"script_fields\": >> {}", >>> - "\"sort\": [ {\"state\": \"asc\"}, {\"id\": \"asc\"}]", >>> - "\"from\": 2", >>> - "\"size\": 3")); >>> - } >>> - >>> - @Test public void testOffsetLimit() { >>> - final String sql = "select \"state\", \"id\" from zips\n" >>> - + "offset 2 fetch next 3 rows only"; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query(sql) >>> - .runs() >>> - .queryContains( >>> - elasticsearchChecker( >>> - "\"from\": 2", >>> - "\"size\": 3", >>> - "\"fields\" : [\"state\", \"id\"], \"script_fields\": >> {}")); >>> - } >>> - >>> - @Test public void testLimit() { >>> - final String sql = "select \"state\", \"id\" from zips\n" >>> - + "fetch next 3 rows only"; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query(sql) >>> - .runs() >>> - .queryContains( >>> - elasticsearchChecker( >>> - "\"size\": 3", >>> - "\"fields\" : [\"state\", \"id\"], \"script_fields\": >> {}")); >>> - } >>> - >>> - @Test public void testFilterSort() { >>> - final String sql = "select * from zips\n" >>> - + "where \"city\" = 'SPRINGFIELD' and \"id\" >= '70000'\n" >>> - + "order by \"state\", \"id\""; >>> - final String explain = "PLAN=ElasticsearchToEnumerableConverter\n" >>> - + " ElasticsearchSort(sort0=[$4], sort1=[$5], dir0=[ASC], >> dir1=[ASC])\n" >>> - + " ElasticsearchProject(city=[CAST(ITEM($0, >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'), >> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT], >> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0, >> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5) >> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\"])\n" >>> - + " ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0, >> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\", 'SPRINGFIELD'), >=(CAST(ITEM($0, >> 'id')):VARCHAR(5) CHARACTER SET \"ISO-8859-1\" COLLATE >> \"ISO-8859-1$en_US$primary\", '70000'))])\n" >>> - + " ElasticsearchTableScan(table=[[elasticsearch_raw, >> zips]])"; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query(sql) >>> - .returnsOrdered( >>> - "city=SPRINGFIELD; longitude=-92.54567; latitude=35.274879; >> pop=752; state=AR; id=72157", >>> - "city=SPRINGFIELD; longitude=-102.617322; >> latitude=37.406727; pop=1992; state=CO; id=81073", >>> - "city=SPRINGFIELD; longitude=-90.577479; >> latitude=30.415738; pop=5597; state=LA; id=70462", >>> - "city=SPRINGFIELD; longitude=-123.015259; >> latitude=44.06106; pop=32384; state=OR; id=97477", >>> - "city=SPRINGFIELD; longitude=-122.917108; >> latitude=44.056056; pop=27521; state=OR; id=97478") >>> - .queryContains( >>> - elasticsearchChecker("\"query\" : >> {\"constant_score\":{\"filter\":{\"bool\":" >>> - + >> "{\"must\":[{\"term\":{\"city\":\"springfield\"}},{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}", >>> - "\"fields\" : [\"city\", \"pop\", \"state\", \"id\"], >> \"script_fields\": {\"longitude\":{\"script\":\"_source.loc[0]\"}, >> \"latitude\":{\"script\":\"_source.loc[1]\"}}", >>> - "\"sort\": [ {\"state\": \"asc\"}, {\"id\": \"asc\"}]")) >>> - .explainContains(explain); >>> - } >>> - >>> - @Test public void testFilterSortDesc() { >>> - final String sql = "select * from zips\n" >>> - + "where \"pop\" BETWEEN 20000 AND 20100\n" >>> - + "order by \"state\" desc, \"pop\""; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query(sql) >>> - .limit(4) >>> - .returnsOrdered( >>> - "city=SHERIDAN; longitude=-106.964795; latitude=44.78486; >> pop=20025; state=WY; id=82801", >>> - "city=MOUNTLAKE TERRAC; longitude=-122.304036; >> latitude=47.793061; pop=20059; state=WA; id=98043", >>> - "city=FALMOUTH; longitude=-77.404537; latitude=38.314557; >> pop=20039; state=VA; id=22405", >>> - "city=FORT WORTH; longitude=-97.318409; latitude=32.725551; >> pop=20012; state=TX; id=76104"); >>> - } >>> - >>> - @Test public void testFilterRedundant() { >>> - final String sql = "select * from zips\n" >>> - + "where \"state\" > 'CA' and \"state\" < 'AZ' and \"state\" = >> 'OK'"; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query(sql) >>> - .runs() >>> - .queryContains( >>> - elasticsearchChecker("" >>> - + "\"query\" : >> {\"constant_score\":{\"filter\":{\"bool\":" >>> - + "{\"must\":[{\"term\":{\"state\":\"ok\"}}]}}}}", >>> - "\"fields\" : [\"city\", \"pop\", \"state\", \"id\"], >> \"script_fields\": {\"longitude\":{\"script\":\"_source.loc[0]\"}, >> \"latitude\":{\"script\":\"_source.loc[1]\"}}")); >>> - } >>> - >>> - @Test public void testInPlan() { >>> - final String[] searches = { >>> - "\"query\" : >> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":" >>> - + >> "[{\"bool\":{\"must\":[{\"term\":{\"pop\":20012}}]}},{\"bool\":{\"must\":[{\"term\":" >>> - + "{\"pop\":15590}}]}}]}}}}", >>> - "\"fields\" : [\"city\", \"pop\", \"state\", \"id\"], >> \"script_fields\": {\"longitude\":{\"script\":\"_source.loc[0]\"}, >> \"latitude\":{\"script\":\"_source.loc[1]\"}}" >>> - }; >>> - CalciteAssert.that() >>> - .enable(enabled()) >>> - .with(ZIPS) >>> - .query("select * from zips where \"pop\" in (20012, 15590)") >>> - .returnsUnordered( >>> - "city=COVINA; longitude=-117.884285; latitude=34.08596; >> pop=15590; state=CA; id=91723", >>> - "city=ARL
