[GitHub] drill pull request #1126: DRILL-6179: Added pcapng-format support
GitHub user Vlad-Storona opened a pull request: https://github.com/apache/drill/pull/1126 DRILL-6179: Added pcapng-format support See DRILL-6179 for details. https://issues.apache.org/jira/browse/DRILL-6179 You can merge this pull request into a Git repository by running: $ git pull https://github.com/mapr-demos/drill pcapng_dev Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1126.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1126 commit 043988c0a12bca3288f8ac49384ba6d1584fa159 Author: Vlad Storona <vstorona@...> Date: 2018-01-30T12:55:04Z DRILL-6179: Added pcapng-format support ---
[GitHub] drill pull request #1102: DRILL-6119: The OpenTSDB storage plugin is not inc...
GitHub user Vlad-Storona opened a pull request: https://github.com/apache/drill/pull/1102 DRILL-6119: The OpenTSDB storage plugin is not included in the Drill distribution You can merge this pull request into a Git repository by running: $ git pull https://github.com/mapr-demos/drill master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1102.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1102 commit 065a147dfbb1edf8305deee750fd08546c8fd8e4 Author: Vlad Storona <vstorona@...> Date: 2018-01-30T12:55:04Z DRILL-6119: The OpenTSDB storage plugin is not included in the Drill distribution ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957508
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java
---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+this.db = db;
+this.name = name;
+setupStructure();
+ }
+
+ private void setupStructure() {
+columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(),
OpenTSDBTypes.STRING));
+columns.add(new ColumnDTO(TIMESTAMP.toString(),
OpenTSDBTypes.TIMESTAMP));
+columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(),
OpenTSDBTypes.DOUBLE));
+columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List
+ */
+ public List getColumns() {
+return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957504
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java
---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
--- End diff --
Sure, I will rename it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957506
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java
---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+ /**
+ * Parse FROM parameters to Map representation
+ *
+ * @param rowData with this syntax (metric=warp.speed.test)
+ * @return Map with params key: metric, value: warp.speed.test
+ */
+ public static Map<String, String> parseFromRowData(String rowData) {
+String FROMRowData = rowData.replaceAll("[()]", "");
--- End diff --
I will rename it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957511
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java
---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request
to DB.
+ *
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result
set.
+ */
+public class DBQuery {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(DBQuery.class);
+ /**
+ * The start time for the query. This can be a relative or absolute
timestamp.
+ */
+ private String start;
+ /**
+ * An end time for the query. If not supplied, the TSD will assume the
local system time on the server.
+ * This may be a relative or absolute timestamp. This param is optional,
and if it isn't specified we will send null
+ * to the db in this field, but in this case db will assume the local
system time on the server.
+ */
+ private String end;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set queries;
+
+ private DBQuery(Builder builder) {
+this.start = builder.start;
+this.end = builder.end;
+this.queries = builder.queries;
+ }
+
+ public String getStart() {
+return start;
+ }
+
+ public String getEnd() {
+return end;
+ }
+
+ public Set getQueries() {
+return queries;
+ }
+
+ public static class Builder {
+
+private String start;
+private String end;
+private Set queries = new HashSet<>();
+
+public Builder() {
+}
+
+public Builder setStartTime(String startTime) {
+ if (startTime == null) {
+throw UserException.validationError()
+.message("start param must be specified")
+.build(log);
+ }
+ this.start = startTime;
+ return this;
+}
+
+public Builder setEndTime(String endTime) {
+ this.end = endTime;
+ return this;
+}
+
+public Builder setQueries(Set queries) {
+ if (queries.isEmpty()) {
+throw UserException.validationError()
+.message("Required params such as metric, aggregator
weren't specified. " +
+"Add this params to the query")
--- End diff --
Ok
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request: https://github.com/apache/drill/pull/774#discussion_r148957497 --- Diff: contrib/storage-opentsdb/README.md --- @@ -0,0 +1,64 @@ +# drill-storage-openTSDB + +Implementation of TSDB storage plugin. Plugin uses REST API to work with TSDB. + +For more information about openTSDB follow this link <http://opentsdb.net> + +There is list of required params: + +* metric - The name of a metric stored in the db. + +* start - The start time for the query. This can be a relative or absolute timestamp. + +* aggregator - The name of an aggregation function to use. + +optional param is: + +* downsample - An optional downsampling function to reduce the amount of data returned. + +* end - An end time for the query. If not supplied, the TSD will assume the local system time on the server. +This may be a relative or absolute timestamp. This param is optional, and if it isn't specified we will send null +to the db in this field, but in this case db will assume the local system time on the server. + +List of supported aggregators + +<http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html> + +List of supported time + +<http://opentsdb.net/docs/build/html/user_guide/query/dates.html> + +Params must be specified in FROM clause of the query separated by commas. For example + +`openTSDB.(metric=metric_name, start=4d-ago, aggregator=sum)` + +Supported queries for now are listed below: + +``` +USE openTSDB +``` + +``` +SHOW tables +``` +Will print available metrics. Max number of the printed results is a Integer.MAX value + +``` +SELECT * FROM openTSDB. `(metric=warp.speed.test, start=47y-ago, aggregator=sum)` +``` +Return aggregated elements from `warp.speed.test` table since 47y-ago + +``` +SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg, start=47y-ago)` +``` +Return aggregated elements from `warp.speed.test` table + +``` +SELECT `timestamp`, sum(`aggregated value`) FROM openTSDB.`(metric=warp.speed.test, aggregator=avg, start=47y-ago)` GROUP BY `timestamp` --- End diff -- Ok, I will add it. ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957500
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java
---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public interface Constants {
+ /**
+ * openTSDB required constants for API call
+ */
+ public static final String DEFAULT_TIME = "47y-ago";
--- End diff --
Yes, I remember that, but when we execute the query like `show tables;`
drill must create the schema of the table. To create a schema we need to send a
request to the db, but for this, we need the required params. But from this
query, we can get only the metric name. That is why I use this default params.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957509
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java
---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static
org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+this.db = db;
+this.name = name;
+setupStructure();
+ }
+
+ private void setupStructure() {
+columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(),
OpenTSDBTypes.STRING));
+columns.add(new ColumnDTO(TIMESTAMP.toString(),
OpenTSDBTypes.TIMESTAMP));
+columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(),
OpenTSDBTypes.DOUBLE));
+columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List
+ */
+ public List getColumns() {
+return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
+HashMap<String, String> params = new HashMap<>();
+params.put(METRIC_PARAM, getValidTableName(name));
+params.put(AGGREGATOR_PARAM, SUM_AGGREGATOR);
+params.put(TIME_PARAM, DEFAULT_TIME);
--- End diff --
Explained this in the comment above.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957514
--- Diff:
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestOpenTSDBPlugin.java
---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson;
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WTIHOUT_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITHOUT_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITH_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.REQUEST_TO_NONEXISTENT_METRIC;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_NAME_REQUEST;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_REQUEST;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITHOUT_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS;
+
+public class TestOpenTSDBPlugin extends PlanTestBase {
+
+ protected static OpenTSDBStoragePlugin storagePlugin;
+ protected static OpenTSDBStoragePluginConfig storagePluginConfig;
+
+ @Rule
+ public WireMockRule wireMockRule = new WireMockRule(1);
+
+ @BeforeClass
+ public static void addTestDataToDB() throws Exception {
--- End diff --
Ok
---
[GitHub] drill issue #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on the issue: https://github.com/apache/drill/pull/774 @arina-ielchiieva Thanks for the review. I updated the PR. Please review updated changes. ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148628710
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBDatabaseSchema.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import
org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory.OpenTSDBTables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.Set;
+
+public class OpenTSDBDatabaseSchema extends AbstractSchema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBDatabaseSchema.class);
+
+ private final OpenTSDBTables schema;
+ private final Set tableNames;
+
+ private final Map<String, DrillTable> drillTables = Maps.newHashMap();
+
+ public OpenTSDBDatabaseSchema(Set tableList, OpenTSDBTables
schema,
+String name) {
+super(schema.getSchemaPath(), name);
+this.schema = schema;
+this.tableNames = Sets.newHashSet(tableList);
--- End diff --
I rewrited `OpenTSDBSchemaFactory`, and for that moment
`OpenTSDBDatabaseSchema` not needed.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148628656
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
--- End diff --
It must be shared among several queries.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148628583
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDB.java
---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import retrofit2.Call;
+import retrofit2.http.Body;
+import retrofit2.http.GET;
+import retrofit2.http.POST;
+import retrofit2.http.Query;
+
+import java.util.Set;
+
+/**
+ * Client for API requests to openTSDB
+ */
+public interface OpenTSDB {
+
+ /**
+ * Used for getting all metrics names from openTSDB
+ *
+ * @return Set with all tables names
+ */
+ @GET("api/suggest?type=metrics=999")
--- End diff --
I do not find another way of getting all the metrics from openTSDB. And I
don't know in what place we can configure this value. I suppose we can set the
max value for this parameter and get all the metrics.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569640
--- Diff: exec/java-exec/src/test/resources/drill-module.conf ---
@@ -10,7 +10,7 @@ drill: {
packages += "org.apache.drill.exec.testing",
packages += "org.apache.drill.exec.rpc.user.security.testing"
}
- test.query.printing.silent : false,
+ test.query.printing.silent : false,
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request: https://github.com/apache/drill/pull/774#discussion_r148569604 --- Diff: distribution/pom.xml --- @@ -190,11 +190,21 @@ org.apache.drill.contrib + drill-opentsdb-storage --- End diff -- This is merge mistake. I will solve this problem. ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569566
--- Diff:
contrib/storage-opentsdb/src/test/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+ "storage": {
+openTSDB: {
+ type: "openTSDB",
+ connection: "localhost:8089",
--- End diff --
Sure.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569546
--- Diff:
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestOpenTSDBPlugin.java
---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson;
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WTIHOUT_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITHOUT_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITH_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.REQUEST_TO_NONEXISTENT_METRIC;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_NAME_REQUEST;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_REQUEST;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITHOUT_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS;
+
+public class TestOpenTSDBPlugin extends BaseTestQuery {
--- End diff --
This type of query unsupported in this realization.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request: https://github.com/apache/drill/pull/774#discussion_r148569587 --- Diff: contrib/storage-opentsdb/src/test/resources/logback.xml --- @@ -0,0 +1,64 @@ + --- End diff -- Ok, I will remove this file. ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569487
--- Diff:
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java
---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569460
--- Diff:
contrib/storage-opentsdb/src/main/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+ "storage": {
+openTSDB: {
+ type: "openTSDB",
+ connection: "1.2.3.4:4242",
+ enabled: true
--- End diff --
For now, a user cannot choose what type of connection will be used by
drill. But it must be configurable. I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request: https://github.com/apache/drill/pull/774#discussion_r148569621 --- Diff: distribution/src/assemble/bin.xml --- @@ -99,7 +99,9 @@ org.apache.drill.contrib:drill-format-mapr org.apache.drill.contrib:drill-jdbc-storage org.apache.drill.contrib:drill-kudu-storage +org.apache.drill.contrib:drill-opentsdb-storage --- End diff -- It is the same problem as in pom file. I will solve it. ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569508
--- Diff:
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java
---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
+
+ static final String SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS = "[{" +
--- End diff --
Sure.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569476
--- Diff:
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestBase.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.junit.Assert;
+
+import java.util.List;
+
+class TestBase extends PlanTestBase {
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569335
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+private final Map<String, OpenTSDBDatabaseSchema> schemaMap =
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+ super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+ Set tables;
+ if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this,
name));
+ }
+ return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+ return Collections.emptySet();
--- End diff --
Yes, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569425
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+private final Map<String, OpenTSDBDatabaseSchema> schemaMap =
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+ super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+ Set tables;
+ if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this,
name));
+ }
+ return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+ return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+return new DrillOpenTSDBTable(schemaName, plugin, new
Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
+return null;
+ }
+}
+
+@Override
+public Set getTableNames() {
+ return plugin.getClient().getAllTableNames();
+}
+
+@Override
+public CreateTableEntry createNewTable(final String tableName,
List partitionColumns) {
+ return null;
+}
+
+@Override
+public void dropTable(String tableName) {
+}
+
+@Override
+public boolean isMutable() {
--- End diff --
Yes, it is a mistake, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569297
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
--- End diff --
Ok, good idea.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569383
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+private final Map<String, OpenTSDBDatabaseSchema> schemaMap =
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+ super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+ Set tables;
+ if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this,
name));
+ }
+ return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+ return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+return new DrillOpenTSDBTable(schemaName, plugin, new
Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
+return null;
+ }
+}
+
+@Override
+public Set getTableNames() {
+ return plugin.getClient().getAllTableNames();
+}
+
+@Override
+public CreateTableEntry createNewTable(final String tableName,
List partitionColumns) {
+ return null;
--- End diff --
Ok, I will remove this overriding of the method.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569362
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+private final Map<String, OpenTSDBDatabaseSchema> schemaMap =
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+ super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+ Set tables;
+ if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this,
name));
+ }
+ return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+ return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+return new DrillOpenTSDBTable(schemaName, plugin, new
Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
--- End diff --
I think we must throw a meaningful exception to the user. I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569320
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+private final Map<String, OpenTSDBDatabaseSchema> schemaMap =
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+ super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+ Set tables;
--- End diff --
Ok, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569400
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+private final Map<String, OpenTSDBDatabaseSchema> schemaMap =
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+ super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+ Set tables;
+ if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this,
name));
+ }
+ return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+ return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+return new DrillOpenTSDBTable(schemaName, plugin, new
Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
+return null;
+ }
+}
+
+@Override
+public Set getTableNames() {
+ return plugin.getClient().getAllTableNames();
+}
+
+@Override
+public CreateTableEntry createNewTable(final String tableName,
List partitionColumns) {
+ return null;
+}
+
+@Override
+public void dropTable(String tableName) {
--- End diff --
Ok, I will remove this overriding of the method.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569232
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
--- End diff --
Yes, missed it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569173
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set getTablesFromDB() {
+return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set getAllTableNames() {
+return getTableNames();
+ }
+
+ @Override
+ public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+}
+ }
+}
+return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+} else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+}
+ }
+
+ private Set getAllMetricsByTags() {
+try {
+ return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+}
+ }
+
+ private Set getTableNames() {
+try {
+ return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+}
+ }
+
+ private Set getAllTablesWithSpecialTag(DBQuery base) throws
IOException {
+return client.getTables(base).execute().body();
+ }
+
+ private Set getAllMetricsFromDBByTags() throws IOException {
+Map<String, String> tags = new HashMap<>();
+
+Query
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569123
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set getTablesFromDB() {
+return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set getAllTableNames() {
+return getTableNames();
+ }
+
+ @Override
+ public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+}
+ }
+}
+return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+} else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+}
+ }
+
+ private Set getAllMetricsByTags() {
+try {
+ return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+}
+ }
+
+ private Set getTableNames() {
+try {
+ return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+}
+ }
+
+ private Set getAllTablesWithSpecialTag(DBQuery base) throws
IOException {
+return client.getTables(base).execute().body();
+ }
+
+ private Set getAllMetricsFromDBByTags() throws IOException {
+Map<String, String> tags = new HashMap<>();
+
+Query
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569087
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set getTablesFromDB() {
+return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set getAllTableNames() {
+return getTableNames();
+ }
+
+ @Override
+ public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+}
+ }
+}
+return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+} else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+}
+ }
+
+ private Set getAllMetricsByTags() {
+try {
+ return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+}
+ }
+
+ private Set getTableNames() {
+try {
+ return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+}
+ }
+
+ private Set getAllTablesWithSpecialTag(DBQuery base) throws
IOException {
+return client.getTables(base).execute().body();
+ }
+
+ private Set getAllMetricsFromDBByTags() throws IOException {
+Map<String, String> tags = new HashMap<>();
+
+Query
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568996
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set getTablesFromDB() {
+return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set getAllTableNames() {
+return getTableNames();
+ }
+
+ @Override
+ public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+}
+ }
+}
+return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+} else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+}
+ }
+
+ private Set getAllMetricsByTags() {
+try {
+ return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+}
+ }
+
+ private Set getTableNames() {
+try {
+ return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
--- End diff --
This is really bad practice, sorry for that, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568915
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
--- End diff --
Sure, it must be final.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568965
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set getTablesFromDB() {
+return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set getAllTableNames() {
+return getTableNames();
+ }
+
+ @Override
+ public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+}
+ }
+}
+return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+} else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+}
+ }
+
+ private Set getAllMetricsByTags() {
+try {
+ return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+ logIOException(e);
--- End diff --
No, it is the critical exception. I will add exception throwing with the
detailed message.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568941
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set getTablesFromDB() {
--- End diff --
Sure, I will add comments for methods in the Service interface.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568893
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java
---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static
org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ *
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+ /**
+ * The name of an aggregation function to use.
+ */
+ private String aggregator;
+ /**
+ * The name of a metric stored in the system
+ */
+ private String metric;
+ /**
+ * Whether or not the data should be converted into deltas before
returning.
+ * This is useful if the metric is a continuously incrementing counter
+ * and you want to view the rate of change between data points.
+ */
+ private String rate;
+ /**
+ * An optional downsampling function to reduce the amount of data
returned.
+ */
+ private String downsample;
+ /**
+ * To drill down to specific timeseries or group results by tag,
+ * supply one or more map values in the same format as the query string.
+ */
+ private Map<String, String> tags;
+
+ private Query(Builder builder) {
+this.aggregator = builder.aggregator;
+this.metric = builder.metric;
+this.rate = builder.rate;
+this.downsample = builder.downsample;
+this.tags = builder.tags;
+ }
+
+ public String getAggregator() {
+return aggregator;
+ }
+
+ public String getMetric() {
+return metric;
+ }
+
+ public String getRate() {
+return rate;
+ }
+
+ public String getDownsample() {
+return downsample;
+ }
+
+ public Map<String, String> getTags() {
+return tags;
+ }
+
+ public static class Builder {
+
+private String aggregator = SUM_AGGREGATOR;
+private String metric;
+private String rate;
+private String downsample;
+private Map<String, String> tags = new HashMap<>();
+
+public Builder(String metric) {
+ this.metric = metric;
+}
+
+public Builder setAggregator(String aggregator) {
+ if (aggregator != null) {
+this.aggregator = aggregator;
+ }
+ return this;
+}
+
+public Builder setMetric(String metric) {
+ this.metric = metric;
+ return this;
+}
+
+public Builder setRate(String rate) {
+ if (rate != null) {
--- End diff --
Ok, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568843
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java
---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request
to DB.
+ *
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result
set.
+ */
+public class DBQuery {
+
+ /**
+ * The start time for the query. This can be a relative or absolute
timestamp.
+ */
+ private String start;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set queries;
+
+ private DBQuery(Builder builder) {
+this.start = builder.start;
+this.queries = builder.queries;
+ }
+
+ public String getStart() {
+return start;
+ }
+
+ public Set getQueries() {
+return queries;
+ }
+
+ public static class Builder {
+
+private String start = DEFAULT_TIME;
+private Set queries = new HashSet<>();
--- End diff --
Yes, good idea, I will add this type of check to the `Builder`.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568871
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java
---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static
org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ *
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+ /**
+ * The name of an aggregation function to use.
+ */
+ private String aggregator;
+ /**
+ * The name of a metric stored in the system
+ */
+ private String metric;
+ /**
+ * Whether or not the data should be converted into deltas before
returning.
+ * This is useful if the metric is a continuously incrementing counter
+ * and you want to view the rate of change between data points.
+ */
+ private String rate;
+ /**
+ * An optional downsampling function to reduce the amount of data
returned.
+ */
+ private String downsample;
+ /**
+ * To drill down to specific timeseries or group results by tag,
+ * supply one or more map values in the same format as the query string.
+ */
+ private Map<String, String> tags;
+
+ private Query(Builder builder) {
+this.aggregator = builder.aggregator;
+this.metric = builder.metric;
+this.rate = builder.rate;
+this.downsample = builder.downsample;
+this.tags = builder.tags;
+ }
+
+ public String getAggregator() {
+return aggregator;
+ }
+
+ public String getMetric() {
+return metric;
+ }
+
+ public String getRate() {
+return rate;
+ }
+
+ public String getDownsample() {
+return downsample;
+ }
+
+ public Map<String, String> getTags() {
+return tags;
+ }
+
+ public static class Builder {
+
+private String aggregator = SUM_AGGREGATOR;
--- End diff --
Sure, I will remove support for these type of queries.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568822
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java
---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request
to DB.
+ *
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result
set.
+ */
+public class DBQuery {
+
+ /**
+ * The start time for the query. This can be a relative or absolute
timestamp.
+ */
+ private String start;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set queries;
+
+ private DBQuery(Builder builder) {
+this.start = builder.start;
+this.queries = builder.queries;
+ }
+
+ public String getStart() {
+return start;
+ }
+
+ public Set getQueries() {
+return queries;
+ }
+
+ public static class Builder {
+
+private String start = DEFAULT_TIME;
--- End diff --
Sure, I will change this behavior, and this parameter will be required for
all select queries.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568797
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDBTypes.java
---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+/**
+ * Types in openTSDB records,
+ * used for converting openTSDB data to Sql representation
+ */
+public enum OpenTSDBTypes {
+ STRING,
+ DOUBLE,
+ TIMESTAMP,
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568766
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java
---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+ /**
+ * Parse FROM parameters to Map representation
+ *
+ * @param rowData with this syntax (metric=warp.speed.test)
+ * @return Map with params key: metric, value: warp.speed.test
+ */
+ public static Map<String, String> parseFROMRowData(String rowData) {
--- End diff --
I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568739
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java
---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+ @JsonProperty
+ public final OpenTSDBStoragePluginConfig storage;
+
+ private final List columns;
+ private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+ private final List tabletScanSpecList;
+
+ @JsonCreator
+ public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig
storage,
+ @JsonProperty("tabletScanSpecList")
LinkedList tabletScanSpecList,
+ @JsonProperty("columns") List
columns) throws ExecutionSetupException {
+super((String) null);
+openTSDBStoragePlugin = (OpenTSDBStoragePlugin)
registry.getPlugin(storage);
+this.tabletScanSpecList = tabletScanSpecList;
+this.storage = (OpenTSDBStoragePluginConfig) storage;
+this.columns = columns;
+ }
+
+ public OpenTSDBSubScan(OpenTSDBStoragePlugin plugin,
OpenTSDBStoragePluginConfig config,
+ List tabletInfoList,
List columns) {
+super((String) null);
+openTSDBStoragePlugin = plugin;
+storage = config;
+this.tabletScanSpecList = tabletInfoList;
+this.columns = columns;
+ }
+
+ @Override
+ public int getOperatorType() {
+return 0;
+ }
+
+ @Override
+ public boolean isExecutable() {
+return false;
+ }
+
+ @Override
+ public PhysicalOperator getNewWithChildren(List
children) throws ExecutionSetupException {
+Preconditions.checkArgument(children.isEmpty());
+return new OpenTSDBSubScan(openTSDBStoragePlugin, storage,
tabletScanSpecList, columns);
+ }
+
+ @Override
+ public Iterator iterator() {
+return Collections.emptyIterator();
+ }
+
+ @Override
+ public <T, X, E extends Throwable> T accept(PhysicalVisitor<T, X, E>
physicalVisitor, X value) throws E {
+return physicalVisitor.visitSubScan(this, value);
+ }
+
+ public List getColumns() {
+return columns;
+ }
+
+ public List getTabletScanSpecList() {
+return tabletScanSpecList;
+ }
+
+ @JsonIgnore
+ public OpenTSDBStoragePlugin getStorageEngine() {
+return openTSDBStoragePlugin;
+ }
+
+ @JsonIgnore
+ public OpenTSDBStoragePluginConfig getStorageConfig() {
--- End diff --
I will fix it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567893
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator tableIterator;
+ private OutputMutator output;
+ private ImmutableList projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec
subScanSpec,
+ List projectedColumns) throws
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+try {
+ return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+}
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
+return super.isSkipQuery();
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ static {
+TYPES = ImmutableMap.<OpenTSDBTypes, MinorType>builder()
+.put(OpenTSDBTypes.STRING, Minor
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568059
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePluginConfig.java
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.logical.StoragePluginConfigBase;
+
+import java.io.IOException;
+import java.util.Objects;
+
+@JsonTypeName(OpenTSDBStoragePluginConfig.NAME)
+public class OpenTSDBStoragePluginConfig extends StoragePluginConfigBase {
+
+ public static final String NAME = "openTSDB";
+
+ private final String connection;
+
+ @JsonCreator
+ public OpenTSDBStoragePluginConfig(@JsonProperty("connection") String
connection) throws IOException {
+this.connection = connection;
--- End diff --
Yes, we should. I will add null check.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568016
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java
---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration,
DrillbitContext context, String name) throws IOException {
+this.context = context;
+this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+this.engineConfig = configuration;
+this.db = new ServiceImpl("http://; + configuration.getConnection());
+ }
+
+ @Override
+ public void start() throws IOException {
--- End diff --
Sure, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568092
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java
---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+ @JsonProperty
+ public final OpenTSDBStoragePluginConfig storage;
+
+ private final List columns;
+ private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+ private final List tabletScanSpecList;
+
+ @JsonCreator
+ public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig
storage,
+ @JsonProperty("tabletScanSpecList")
LinkedList tabletScanSpecList,
+ @JsonProperty("columns") List
columns) throws ExecutionSetupException {
+super((String) null);
+openTSDBStoragePlugin = (OpenTSDBStoragePlugin)
registry.getPlugin(storage);
+this.tabletScanSpecList = tabletScanSpecList;
+this.storage = (OpenTSDBStoragePluginConfig) storage;
--- End diff --
Thanks. I will change this casting.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568031
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java
---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration,
DrillbitContext context, String name) throws IOException {
+this.context = context;
+this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+this.engineConfig = configuration;
+this.db = new ServiceImpl("http://; + configuration.getConnection());
+ }
+
+ @Override
+ public void start() throws IOException {
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ @Override
+ public boolean supportsRead() {
+return true;
+ }
+
+ @Override
+ public OpenTSDBStoragePluginConfig getConfig() {
+return engineConfig;
+ }
+
+ @Override
+ public OpenTSDBGroupScan getPhysicalScan(String userName, JSONOptions
selection) throws IOException {
+OpenTSDBScanSpec scanSpec = selection.getListWith(new ObjectMapper(),
new TypeReference() {
+});
+return new OpenTSDBGroupScan(this, scanSpec, null);
+ }
+
+ @Override
+ public boolean supportsWrite() {
+return true;
--- End diff --
Ok, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567959
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java
---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration,
DrillbitContext context, String name) throws IOException {
+this.context = context;
+this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+this.engineConfig = configuration;
+this.db = new ServiceImpl("http://; + configuration.getConnection());
--- End diff --
Thank for this remark, I will move this choice to the config's.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567934
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBScanSpec.java
---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class OpenTSDBScanSpec {
+
+ private final String tableName;
+
+ @JsonCreator
+ public OpenTSDBScanSpec(@JsonProperty("tableName") String tableName) {
+this.tableName = tableName;
+ }
+
+ public String getTableName() {
+return tableName;
+ }
--- End diff --
Sure, I will add `toString()`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567854
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator tableIterator;
+ private OutputMutator output;
+ private ImmutableList projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec
subScanSpec,
+ List projectedColumns) throws
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+try {
+ return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+}
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
--- End diff --
I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567870
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator tableIterator;
+ private OutputMutator output;
+ private ImmutableList projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec
subScanSpec,
+ List projectedColumns) throws
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+try {
+ return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+}
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
+return super.isSkipQuery();
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ static {
+TYPES = ImmutableMap.<OpenTSDBTypes, MinorType>builder()
+.put(OpenTSDBTypes.STRING, Minor
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567835
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator tableIterator;
+ private OutputMutator output;
+ private ImmutableList projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec
subScanSpec,
+ List projectedColumns) throws
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+try {
+ return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+ log.info(e.toString());
--- End diff --
Sure, I will replace it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567715
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List columns;
+ private List openTSDBWorkList = Lists.newArrayList();
+ private List affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec")
OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage")
OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List
columns,
+ @JacksonInject StoragePluginRegistry
pluginRegistry) throws IOException, ExecutionSetupException {
+this((OpenTSDBStoragePlugin)
pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec,
columns);
+ }
+
+ public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+ OpenTSDBScanSpec scanSpec, List
columns) {
+super((String) null);
+this.storagePlugin = storagePlugin;
+this.storagePluginConfig = storagePlugin.getConfig();
+this.openTSDBScanSpec = scanSpec;
+this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS :
columns;
+init();
+ }
+
+ /**
+ * Private constructor, used for cloning.
+ *
+ * @param that The OpenTSDBGroupScan to clone
+ */
+ private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+super((String) null);
+this.columns = that.columns;
+this.openTSDBScanSpec = that.openTSDBScanSpec;
+this.storagePlugin = that.storagePlugin;
+this.storagePluginConfig = that.storagePl
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567810
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator tableIterator;
+ private OutputMutator output;
+ private ImmutableList projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec
subScanSpec,
+ List projectedColumns) throws
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or
it's empty", subScanSpec.getTableName()));
--- End diff --
Ok, this is really weird behaviour, I will change it. Yes,
`db.getTablesFromDB();` returns all the metrics from db. I will add comments to
the service methods.
Also, I completely agree that the exception handling in this place really
confusing, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567743
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log =
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator tableIterator;
+ private OutputMutator output;
+ private ImmutableList projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec
subScanSpec,
--- End diff --
Ok, I will add access modifier.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567693
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List columns;
+ private List openTSDBWorkList = Lists.newArrayList();
+ private List affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec")
OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage")
OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List
columns,
+ @JacksonInject StoragePluginRegistry
pluginRegistry) throws IOException, ExecutionSetupException {
+this((OpenTSDBStoragePlugin)
pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec,
columns);
+ }
+
+ public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+ OpenTSDBScanSpec scanSpec, List
columns) {
+super((String) null);
+this.storagePlugin = storagePlugin;
+this.storagePluginConfig = storagePlugin.getConfig();
+this.openTSDBScanSpec = scanSpec;
+this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS :
columns;
+init();
+ }
+
+ /**
+ * Private constructor, used for cloning.
+ *
+ * @param that The OpenTSDBGroupScan to clone
+ */
+ private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+super((String) null);
+this.columns = that.columns;
+this.openTSDBScanSpec = that.openTSDBScanSpec;
+this.storagePlugin = that.storagePlugin;
+this.storagePluginConfig = that.storagePl
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567620
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List columns;
+ private List openTSDBWorkList = Lists.newArrayList();
+ private List affinities;
+
+ private boolean filterPushedDown = false;
--- End diff --
No, this variable is not required, I will delete it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567658
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List columns;
+ private List openTSDBWorkList = Lists.newArrayList();
+ private List affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec")
OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage")
OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List
columns,
--- End diff --
Thank for this advice, I will add getters.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567585
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/DrillOpenTSDBTable.java
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+
+import java.util.List;
+
+public class DrillOpenTSDBTable extends DynamicDrillTable {
+
+ private final Schema schema;
+
+ public DrillOpenTSDBTable(String storageEngineName,
OpenTSDBStoragePlugin plugin, Schema schema, OpenTSDBScanSpec scanSpec) {
+super(plugin, storageEngineName, scanSpec);
+this.schema = schema;
+ }
+
+ @Override
+ public RelDataType getRowType(final RelDataTypeFactory typeFactory) {
+List names = Lists.newArrayList();
+List types = Lists.newArrayList();
+convertToRelDataType(typeFactory, names, types);
+return typeFactory.createStructType(types, names);
+ }
+
+ private void convertToRelDataType(RelDataTypeFactory typeFactory,
List names, List types) {
+for (ColumnDTO column : schema.getColumns()) {
+ names.add(column.getColumnName());
+ RelDataType type = getSqlTypeFromOpenTSDBType(typeFactory,
column.getColumnType());
+ type = typeFactory.createTypeWithNullability(type,
column.isNullable());
+ types.add(type);
+}
+ }
+
+ private RelDataType getSqlTypeFromOpenTSDBType(RelDataTypeFactory
typeFactory, OpenTSDBTypes type) {
+switch (type) {
+ case STRING:
+return typeFactory.createSqlType(SqlTypeName.VARCHAR,
Integer.MAX_VALUE);
+ case DOUBLE:
+return typeFactory.createSqlType(SqlTypeName.DOUBLE);
+ case TIMESTAMP:
+return typeFactory.createSqlType(SqlTypeName.TIMESTAMP);
+ default:
+throw new UnsupportedOperationException("Unsupported type.");
--- End diff --
Sure, I will add information about unsupported types to the exception
message.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request: https://github.com/apache/drill/pull/774#discussion_r148567538 --- Diff: contrib/storage-opentsdb/README.md --- @@ -0,0 +1 @@ +# drill-storage-openTSDB --- End diff -- Sure, I will update README with more detailed info. ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567569
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java
---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public class Constants {
--- End diff --
Ok, I will change it.
---
[GitHub] drill pull request #989: DRILL-5790: PCAP format explicitly opens local file
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/989#discussion_r144286736
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapRecordReader.java
---
@@ -125,8 +129,8 @@ public int next() {
@Override
public void close() throws Exception {
-//buffer = null;
-//in.close();
+in.close();
+fs.close();
--- End diff --
Sure, my mistake, I will remove it.
---
[GitHub] drill pull request #989: DRILL-5790: PCAP format explicitly opens local file
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/989#discussion_r144286706
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/pcap/PcapRecordReader.java
---
@@ -100,14 +104,14 @@ public void setup(final OperatorContext context,
final OutputMutator output) thr
this.output = output;
this.buffer = new byte[10];
- this.in = new FileInputStream(inputPath);
+ this.in = fs.open(pathToFile);
this.decoder = new PacketDecoder(in);
this.validBytes = in.read(buffer);
this.projectedCols = getProjectedColsIfItNull();
setColumns(projectedColumns);
} catch (IOException io) {
throw UserException.dataReadError(io)
- .addContext("File name:", inputPath)
+ .addContext("File name:", pathToFile.toString())
--- End diff --
Thanks, I will replace it.
---
[GitHub] drill pull request #989: DRILL-5790: PCAP format explicitly opens local file
GitHub user Vlad-Storona opened a pull request: https://github.com/apache/drill/pull/989 DRILL-5790: PCAP format explicitly opens local file See DRILL-5790 for details. In the current implementation in master used the local FS and it is working on MapR-FS but does not work on HDFS. Now, for reading files used `org.apache.hadoop.fs.FileSystem`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mapr-demos/drill DRILL-5790 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/989.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #989 commit f968708c0a428ab91fdff8f5e52303b500089b88 Author: Vlad Storona <vstor...@cybervisiontech.com> Date: 2017-10-12T10:16:19Z Fixed problem with explicit opening a local file ---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r143683971
--- Diff:
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePluginConfig.java
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.logical.StoragePluginConfigBase;
+
+import java.io.IOException;
+import java.util.Objects;
+
+@JsonTypeName(OpenTSDBStoragePluginConfig.NAME)
+public class OpenTSDBStoragePluginConfig extends StoragePluginConfigBase {
+
+ public static final String NAME = "openTSDB";
+
+ private final String connection;
+
+ @JsonCreator
+ public OpenTSDBStoragePluginConfig(@JsonProperty("connection") String
connection) throws IOException {
+this.connection = connection;
+ }
+
+ String getConnection() {
--- End diff --
Thanks for your remark. I will fix it.
---
[GitHub] drill issue #831: DRILL-5432: Added pcap-format support
Github user Vlad-Storona commented on the issue: https://github.com/apache/drill/pull/831 As a result of project transferring problems were found out. In java-exec package exists file bootstrap-storage-plugin.json from which drill takes information about supported files formats. But in contrib package, there is no such file. If to transfer pcap-reader to contrib package and not to remove information from bootstrap-storage-plugin.json about pcap format, then there will be JsonMappingException. And if you remove this information from the config file, then drill will can`t find pcap files. Maybe I have not enough info/experience about drill. Maybe you can provide any solution how to handle this ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
