[jira] [Commented] (DRILL-5899) Simple pattern matchers can work with DrillBuf directly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237125#comment-16237125
 ] 

ASF GitHub Bot commented on DRILL-5899:
---

Github user ppadma commented on the issue:

https://github.com/apache/drill/pull/1015
  
@sachouche @paul-rogers Thanks for the review. I updated the PR with review 
comments. 

I made one more change.  Previously, I was copying native memory buffer 
into byte array and using it. Instead, if we go to native memory  directly, 
performance is significantly better. In fact,  it is 3 times faster :-)

Please review updated changes.



> Simple pattern matchers can work with DrillBuf directly
> ---
>
> Key: DRILL-5899
> URL: https://issues.apache.org/jira/browse/DRILL-5899
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
>
> For the 4 simple patterns we have i.e. startsWith, endsWith, contains and 
> constant,, we do not need the overhead of charSequenceWrapper. We can work 
> with DrillBuf directly. This will save us from doing isAscii check and UTF8 
> decoding for each row.
> UTF-8 encoding ensures that no UTF-8 character is a prefix of any other valid 
> character. So, instead of decoding varChar from each row we are processing, 
> encode the patternString once during setup and do raw byte comparison. 
> Instead of bounds checking and reading one byte at a time, we get the whole 
> buffer in one shot and use that for comparison.
> This improved overall performance for filter operator by around 20%. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4990) Use new HDFS API access instead of listStatus to check if users have permissions to access workspace.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236810#comment-16236810
 ] 

ASF GitHub Bot commented on DRILL-4990:
---

Github user ppadma commented on the issue:

https://github.com/apache/drill/pull/652
  
@sohami Sorabh, since you reviewed the original pull request, can you 
please review the updated diffs ?


> Use new HDFS API access instead of listStatus to check if users have 
> permissions to access workspace.
> -
>
> Key: DRILL-4990
> URL: https://issues.apache.org/jira/browse/DRILL-4990
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.8.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Major
>
> For every query, we build the schema tree 
> (runSQL->getPlan->getNewDefaultSchema->getRootSchema). All workspaces in all 
> storage plugins are checked and are added to the schema tree if they are 
> accessible by the user who initiated the query.  For file system plugin, 
> listStatus API is used to check if  the workspace is accessible or not 
> (WorkspaceSchemaFactory.accessible) by the user.  The idea seem to be if the 
> user does not have access to file(s) in the workspace, listStatus will 
> generate an exception and we return false. But, listStatus (which lists all 
> the entries of a directory) is an expensive operation when there are large 
> number of files in the directory. A new API is added in Hadoop 2.6 called 
> access (HDFS-6570) which provides the ability to check if the user has 
> permissions on a file/directory.  Use this new API instead of listStatus. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236791#comment-16236791
 ] 

Timothy Farkas commented on DRILL-5922:
---

[~ben-zvi] The *QueryContext* is managed by the *Foreman* class, which does not 
close the QueryContext. It is also created temporarily for the purpose of 
sending an rpc message in *PlanSplitter.planFragments* and is never closed 
there as well. In both cases the QueryContexts are not part of the list of 
AutoCloseables that we close when a Drillbit is closed. Also this leak doesn't 
appear to just be a shutdown issue, this would also cause memory leaks to occur 
in long running Drillbits.

That is weird that the error doesn't show in embedded mode. It also doesn't 
appear in IntelliJ when the offending unit test is run by itself and only 
appears when all the tests in DrillSeparatePlanningTest are run. Maybe stderr 
just isn't flushed in some cases?

Closing the QueryContext in the two offending locations has fixed the exception 
in the unit test. I'll do more testing to make sure nothing else has broken.

> Intermittent Memory Leaks in the ROOT allocator  
> -
>
> Key: DRILL-5922
> URL: https://issues.apache.org/jira/browse/DRILL-5922
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
>
> This issue was originall found by [~ben-zvi]. I am able to consistently 
> reproduce the error on my laptop by running the following unit test:
> org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery
> {code}
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding 
> child allocators.
> Allocator(ROOT) 0/1048576/10113536/3221225472 (res/actual/peak/limit)
>   child allocators: 1
> Allocator(query:26049b50-0cec-0a92-437c-bbe486e1fcbf) 
> 1048576/0/0/268435456 (res/actual/peak/limit)
>   child allocators: 0
>   ledgers: 0
>   reservations: 0
>   ledgers: 0
>   reservations: 0
>   at 
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) 
> ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at 
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
>  ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205) 
> [classes/:na]
>   at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:157) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:148) 
> [test-classes/:na]
>   at 
> org.apache.drill.exec.DrillSeparatePlanningTest.getFragmentsHelper(DrillSeparatePlanningTest.java:185)
>  [test-classes/:na]
>   at 
> org.apache.drill.exec.DrillSeparatePlanningTest.testMultiMinorFragmentComplexQuery(DrillSeparatePlanningTest.java:108)
>  [test-classes/:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_144]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  [junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  [junit-4.11.jar:na]
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  [junit-4.11.jar:na]
>   at 
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
>  [jmockit-1.3.jar:na]
>   at 
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
>  [jmockit-1.3.jar:na]
>   at 
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
>  [jmockit-1.3.jar:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[na:1.8.0_144]
>   at 
> 

[jira] [Closed] (DRILL-4728) Add support for new metadata fetch APIs

[Chun Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Chang closed DRILL-4728.
-

Automation added.

> Add support for new metadata fetch APIs
> ---
>
> Key: DRILL-4728
> URL: https://issues.apache.org/jira/browse/DRILL-4728
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Metadata
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
>Priority: Major
> Fix For: 1.8.0
>
>
> Please see the doc attached to the parent JIRA DRILL-4714 for details on APIs.
> Add support for following APIs (including {{protobuf}} messages, server 
> handling code and Java client APIs)
> {code}
>List getCatalogs(Filter catalogNameFilter)
>List getSchemas(
>   Filter catalogNameFilter,
>   Filter schemaNameFilter
>)
>List getTables(
>   Filter catalogNameFilter,
>   Filter schemaNameFilter,
>  Filter tableNameFilter
>)
>List getColumns(
>   Filter catalogNameFilter,
>   Filter schemaNameFilter,
>   Filter tableNameFilter,
>   Filter columnNameFilter
>)
> {code}
> Note: native client changes are not going to be included in this patch. Will 
> file a separate JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (DRILL-4730) Update JDBC DatabaseMetaData implementation to use new Metadata APIs

[Chun Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Chang closed DRILL-4730.
-

automation added.

> Update JDBC DatabaseMetaData implementation to use new Metadata APIs
> 
>
> Key: DRILL-4730
> URL: https://issues.apache.org/jira/browse/DRILL-4730
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Client - JDBC
>Reporter: Venki Korukanti
>Assignee: Laurent Goujon
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> DRILL-4728 is going to add support for new metadata APIs. Replace the 
> INFORMATION_SCHEMA queries used to get the metadata with the new APIs 
> provided in Java client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (DRILL-5301) Add server metadata API

[Chun Chang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Chang closed DRILL-5301.
-

Automation added.

> Add server metadata API
> ---
>
> Key: DRILL-5301
> URL: https://issues.apache.org/jira/browse/DRILL-5301
> Project: Apache Drill
>  Issue Type: Improvement
>  Components:  Server, Client - C++, Client - Java, Client - JDBC, 
> Client - ODBC
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> JDBC and ODBC clients exposes lots of metadata regarding server version and 
> support of various parts of the SQL standard.
> Currently the returned information is hardcoded in both clients/drivers which 
> means that the infomation returned is support as of the client version, not 
> the server version.
> Instead, a new method should be provided to the clients to query the actual 
> server support. Support on the client or the server should be optional (for 
> example a client should not use this API if the server doesn't support it and 
> fallback to default values).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5924) native-client: Support user-specified CXX_FLAGS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236700#comment-16236700
 ] 

ASF GitHub Bot commented on DRILL-5924:
---

GitHub user xhochy opened a pull request:

https://github.com/apache/drill/pull/1022

DRILL-5924: native-client: Support user-specified CXX_FLAGS

Also remove the redundant sets of `CMAKE_CXX_FLAGS`. They are set on a 
global scope so that there is no need to reset them more locally.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xhochy/drill native/extend-cxxflags

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1022.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1022


commit 0e96d851c9993ecd6be57fe419ce959b03d4a908
Author: Korn, Uwe 
Date:   2017-10-28T19:45:51Z

DRILL-5924: native-client: Support user-specified CXX_FLAGS




> native-client: Support user-specified CXX_FLAGS
> ---
>
> Key: DRILL-5924
> URL: https://issues.apache.org/jira/browse/DRILL-5924
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - C++
>Reporter: Uwe L. Korn
>
> Currently the build process for the native client overrides the CXX_FLAGS 
> supplied by the user. In some cases we need to pass additional flags, e.g. 
> {{-fpermissive}}, to the build to have it succeed. Thus instead of overriding 
> these flags, they should only be expanded.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5924) native-client: Support user-specified CXX_FLAGS

[Uwe L. Korn (JIRA)]

Uwe L. Korn created DRILL-5924:
--

 Summary: native-client: Support user-specified CXX_FLAGS
 Key: DRILL-5924
 URL: https://issues.apache.org/jira/browse/DRILL-5924
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - C++
Reporter: Uwe L. Korn


Currently the build process for the native client overrides the CXX_FLAGS 
supplied by the user. In some cases we need to pass additional flags, e.g. 
{{-fpermissive}}, to the build to have it succeed. Thus instead of overriding 
these flags, they should only be expanded.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator

[Boaz Ben-Zvi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236656#comment-16236656
 ] 

Boaz Ben-Zvi commented on DRILL-5922:
-

  While this leaks shows consistently in the IDE (Intellij), running the very 
same query in embedded mode never showed this leak (in the log):
{code}
0: jdbc:drill:zk=local> select dir0, sum(o_totalprice) from 
dfs.`/Users/boazben-zvi/IdeaProjects/drill/exec/java-exec/src/test/resources/multilevel/json`
 group by dir0 order by dir0;
+---++
| dir0  |   EXPR$1   |
+---++
| 1994  | 4790829.05 |
| 1995  | 5043329.54999  |
| 1996  | 5717464.22001  |
+---++
3 rows selected (0.349 seconds)
0: jdbc:drill:zk=local> !q
Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
{code}

And the end of the log:
{code}
2017-11-02 14:27:18,443 [26047248-a518-759b-ceeb-57d9e42eae47:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 26047248-a518-759b-ceeb-57d9e42eae47:0:0: 
State to report: FINISHED
2017-11-02 14:27:18,444 [26047248-a518-759b-ceeb-57d9e42eae47:frag:0:0] DEBUG 
o.a.drill.exec.memory.BaseAllocator - closed 
allocator[query:26047248-a518-759b-ceeb-57d9e42eae47].
2017-11-02 14:27:21,314 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[ROOT].
2017-11-02 14:27:21,314 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[ROOT].
2017-11-02 14:27:22,332 [pool-20-thread-2] DEBUG 
o.a.drill.exec.memory.BaseAllocator - closed allocator[rpc:bit-data].
2017-11-02 14:27:22,333 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[rpc:user].
2017-11-02 14:27:22,333 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[rpc:bit-control].
2017-11-02 14:27:22,334 [main] INFO  o.a.drill.exec.compile.CodeCompiler - 
Stats: code gen count: 126, cache miss count: 6, hit rate: 95%
2017-11-02 14:27:22,338 [main] DEBUG o.a.drill.exec.memory.BaseAllocator - 
closed allocator[ROOT].
2017-11-02 14:27:22,338 [main] INFO  o.apache.drill.exec.server.Drillbit - 
Shutdown completed (1023 ms).
2017-11-02 14:27:22,340 [Drillbit-ShutdownHook#0] INFO  
o.apache.drill.exec.server.Drillbit - Received shutdown request.
{code}


> Intermittent Memory Leaks in the ROOT allocator  
> -
>
> Key: DRILL-5922
> URL: https://issues.apache.org/jira/browse/DRILL-5922
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
>
> This issue was originall found by [~ben-zvi]. I am able to consistently 
> reproduce the error on my laptop by running the following unit test:
> org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery
> {code}
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding 
> child allocators.
> Allocator(ROOT) 0/1048576/10113536/3221225472 (res/actual/peak/limit)
>   child allocators: 1
> Allocator(query:26049b50-0cec-0a92-437c-bbe486e1fcbf) 
> 1048576/0/0/268435456 (res/actual/peak/limit)
>   child allocators: 0
>   ledgers: 0
>   reservations: 0
>   ledgers: 0
>   reservations: 0
>   at 
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) 
> ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at 
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
>  ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205) 
> [classes/:na]
>   at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:157) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:148) 
> [test-classes/:na]
>   at 
> org.apache.drill.exec.DrillSeparatePlanningTest.getFragmentsHelper(DrillSeparatePlanningTest.java:185)
>  [test-classes/:na]
>   at 
> org.apache.drill.exec.DrillSeparatePlanningTest.testMultiMinorFragmentComplexQuery(DrillSeparatePlanningTest.java:108)
>  [test-classes/:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_144]
>   at 

[jira] [Commented] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator

[Boaz Ben-Zvi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236653#comment-16236653
 ] 

Boaz Ben-Zvi commented on DRILL-5922:
-

Isn't the QueryContext.close() is actually invoked as an autoCloseable.close () 
:

"2604713b-ea78-61c9-f51b-3c1405d846a3:frag:0:0@5826" daemon prio=10 tid=0x31 
nid=NA runnable
  java.lang.Thread.State: RUNNABLE
  at org.apache.drill.exec.ops.QueryContext.close(QueryContext.java:298)
  at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.suppressingClose(Foreman.java:712)
  at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:762)
  at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:909)
  at 
org.apache.drill.exec.work.foreman.Foreman.access$2600(Foreman.java:116)
  at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:968)
  at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:961)
  at 
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
  at 
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
  at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.addEvent(Foreman.java:963)
  at 
org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:981)
  at 
org.apache.drill.exec.work.foreman.QueryManager.nodeComplete(QueryManager.java:518)
  at 
org.apache.drill.exec.work.foreman.QueryManager.access$100(QueryManager.java:65)
  at 
org.apache.drill.exec.work.foreman.QueryManager$NodeTracker.fragmentComplete(QueryManager.java:482)
  at 
org.apache.drill.exec.work.foreman.QueryManager.fragmentDone(QueryManager.java:155)
  at 
org.apache.drill.exec.work.foreman.QueryManager.access$400(QueryManager.java:65)
  at 
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:545)
  at 
org.apache.drill.exec.rpc.control.WorkEventBus.statusUpdate(WorkEventBus.java:71)
  at 
org.apache.drill.exec.work.fragment.FragmentStatusReporter.sendStatus(FragmentStatusReporter.java:124)
  at 
org.apache.drill.exec.work.fragment.FragmentStatusReporter.stateChanged(FragmentStatusReporter.java:94)
  at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:304)
  at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
  at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
  at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)


> Intermittent Memory Leaks in the ROOT allocator  
> -
>
> Key: DRILL-5922
> URL: https://issues.apache.org/jira/browse/DRILL-5922
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
>
> This issue was originall found by [~ben-zvi]. I am able to consistently 
> reproduce the error on my laptop by running the following unit test:
> org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery
> {code}
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding 
> child allocators.
> Allocator(ROOT) 0/1048576/10113536/3221225472 (res/actual/peak/limit)
>   child allocators: 1
> Allocator(query:26049b50-0cec-0a92-437c-bbe486e1fcbf) 
> 1048576/0/0/268435456 (res/actual/peak/limit)
>   child allocators: 0
>   ledgers: 0
>   reservations: 0
>   ledgers: 0
>   reservations: 0
>   at 
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) 
> ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at 
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
>  ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205) 
> [classes/:na]
>   at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:157) 
> [test-classes/:na]
>   at 
> 

[jira] [Commented] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236569#comment-16236569
 ] 

Timothy Farkas commented on DRILL-5922:
---

It looks like there is something very wrong here. The child allocator that is 
leaked is created inside a QueryContext. The QueryContext closes the child 
allocator when it is closed. However, it looks like QueryContext.close is never 
called anywhere!

> Intermittent Memory Leaks in the ROOT allocator  
> -
>
> Key: DRILL-5922
> URL: https://issues.apache.org/jira/browse/DRILL-5922
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
>
> This issue was originall found by [~ben-zvi]. I am able to consistently 
> reproduce the error on my laptop by running the following unit test:
> org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery
> {code}
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding 
> child allocators.
> Allocator(ROOT) 0/1048576/10113536/3221225472 (res/actual/peak/limit)
>   child allocators: 1
> Allocator(query:26049b50-0cec-0a92-437c-bbe486e1fcbf) 
> 1048576/0/0/268435456 (res/actual/peak/limit)
>   child allocators: 0
>   ledgers: 0
>   reservations: 0
>   ledgers: 0
>   reservations: 0
>   at 
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) 
> ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at 
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
>  ~[classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
> [classes/:na]
>   at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
> [classes/:na]
>   at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205) 
> [classes/:na]
>   at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:157) 
> [test-classes/:na]
>   at 
> org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:148) 
> [test-classes/:na]
>   at 
> org.apache.drill.exec.DrillSeparatePlanningTest.getFragmentsHelper(DrillSeparatePlanningTest.java:185)
>  [test-classes/:na]
>   at 
> org.apache.drill.exec.DrillSeparatePlanningTest.testMultiMinorFragmentComplexQuery(DrillSeparatePlanningTest.java:108)
>  [test-classes/:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_144]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  [junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  [junit-4.11.jar:na]
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  [junit-4.11.jar:na]
>   at 
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
>  [jmockit-1.3.jar:na]
>   at 
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
>  [jmockit-1.3.jar:na]
>   at 
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
>  [jmockit-1.3.jar:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[na:1.8.0_144]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_144]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
>   at 
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
>  [jmockit-1.3.jar:na]
>   at 
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
>  [jmockit-1.3.jar:na]
>   at 
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41) 
> [jmockit-1.3.jar:na]
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
>  [junit-4.11.jar:na]
>   at 
> 

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236561#comment-16236561
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user cgivre commented on a diff in the pull request:

https://github.com/apache/drill/pull/258#discussion_r148653960
  
--- Diff: 
contrib/gis/src/test/java/org/apache/drill/exec/expr/fn/impl/gis/TestGeometryFunctions.java
 ---
@@ -72,4 +72,203 @@ public void testSTWithinQuery() throws Exception {
 .build()
 .run();
   }
+
+  @Test
+  public void testSTXQuery() throws Exception {
--- End diff --

Please include a test for `ST_UnionAggregate()`


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>Priority: Major
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5783) Make code generation in the TopN operator more modular and test it

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas updated DRILL-5783:
--
Description: 
The work for this PR has had several other PRs batched together with it. The 
full description of work is the following:

DRILL-5783

* A unit test is created for the priority queue in the TopN operator
* The code generation classes passed around a completely unused function 
registry reference in some places so I removed it.
* The priority queue had unused parameters for some of its methods so I removed 
them.

DRILL-5841

* There were many many ways in which temporary folders were created in unit 
tests. I have unified the way these folders are created with the 
DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests 
have been updated to use these. The test watchers create temp directories in 
./target//. So all the files generated and used in the context of a test can 
easily be found in the same consistent location.
* This change should fix the sporadic hashagg test failures, as well as 
failures caused by stray files in /tmp

DRILL-5894

* dfs_test is used as a storage plugin throughout the unit tests. This is 
highly confusing and we can just use dfs instead.

*Misc*

* General code cleanup.
* There are many places where String.format is used unnecessarily. The test 
builder methods already use String.format for you when you pass them args. I 
cleaned some of these up.

  was:
The work for this PR has had several other PRs batched together with it. The 
full description of work is the following:

DRILL-5783

* A unit test is created for the priority queue in the TopN operator
* The code generation classes passed around a completely unused function 
registry reference in some places so I removed it.
* The priority queue had unused parameters for some of its methods so I removed 
them.

DRILL-5841

* There were many many ways in which temporary folders were created in unit 
tests. I have unified the way these folders are created with the 
DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests 
have been updated to use these. The test watchers create temp directories in 
./target//. So all the files generated and used in the context of a test can 
easily be found in the same consistent location.
* This change should fix the sporadic hashagg test failures, as well as 
failures caused by stray files in /tmp
* 
DRILL-5894

* dfs_test is used as a storage plugin throughout the unit tests. This is 
highly confusing and we can just use dfs instead.

*Misc*

* General code cleanup.
* There are many places where String.format is used unnecessarily. The test 
builder methods already use String.format for you when you pass them args. I 
cleaned some of these up.


> Make code generation in the TopN operator more modular and test it
> --
>
> Key: DRILL-5783
> URL: https://issues.apache.org/jira/browse/DRILL-5783
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> The work for this PR has had several other PRs batched together with it. The 
> full description of work is the following:
> DRILL-5783
> * A unit test is created for the priority queue in the TopN operator
> * The code generation classes passed around a completely unused function 
> registry reference in some places so I removed it.
> * The priority queue had unused parameters for some of its methods so I 
> removed them.
> DRILL-5841
> * There were many many ways in which temporary folders were created in unit 
> tests. I have unified the way these folders are created with the 
> DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests 
> have been updated to use these. The test watchers create temp directories in 
> ./target//. So all the files generated and used in the context of a test can 
> easily be found in the same consistent location.
> * This change should fix the sporadic hashagg test failures, as well as 
> failures caused by stray files in /tmp
> DRILL-5894
> * dfs_test is used as a storage plugin throughout the unit tests. This is 
> highly confusing and we can just use dfs instead.
> *Misc*
> * General code cleanup.
> * There are many places where String.format is used unnecessarily. The test 
> builder methods already use String.format for you when you pass them args. I 
> cleaned some of these up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5783) Make code generation in the TopN operator more modular and test it

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas updated DRILL-5783:
--
Description: 
The work for this PR has had several other PRs batched together with it. The 
full description of work is the following:

DRILL-5783

* A unit test is created for the priority queue in the TopN operator
* The code generation classes passed around a completely unused function 
registry reference in some places so I removed it.
* The priority queue had unused parameters for some of its methods so I removed 
them.

DRILL-5841

* There were many many ways in which temporary folders were created in unit 
tests. I have unified the way these folders are created with the 
DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests 
have been updated to use these. The test watchers create temp directories in 
./target//. So all the files generated and used in the context of a test can 
easily be found in the same consistent location.
* This change should fix the sporadic hashagg test failures, as well as 
failures caused by stray files in /tmp
* 
DRILL-5894

* dfs_test is used as a storage plugin throughout the unit tests. This is 
highly confusing and we can just use dfs instead.

*Misc*

* General code cleanup.
* There are many places where String.format is used unnecessarily. The test 
builder methods already use String.format for you when you pass them args. I 
cleaned some of these up.

  was:[~timothyfarkas] to add more details here


> Make code generation in the TopN operator more modular and test it
> --
>
> Key: DRILL-5783
> URL: https://issues.apache.org/jira/browse/DRILL-5783
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> The work for this PR has had several other PRs batched together with it. The 
> full description of work is the following:
> DRILL-5783
> * A unit test is created for the priority queue in the TopN operator
> * The code generation classes passed around a completely unused function 
> registry reference in some places so I removed it.
> * The priority queue had unused parameters for some of its methods so I 
> removed them.
> DRILL-5841
> * There were many many ways in which temporary folders were created in unit 
> tests. I have unified the way these folders are created with the 
> DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests 
> have been updated to use these. The test watchers create temp directories in 
> ./target//. So all the files generated and used in the context of a test can 
> easily be found in the same consistent location.
> * This change should fix the sporadic hashagg test failures, as well as 
> failures caused by stray files in /tmp
> * 
> DRILL-5894
> * dfs_test is used as a storage plugin throughout the unit tests. This is 
> highly confusing and we can just use dfs instead.
> *Misc*
> * General code cleanup.
> * There are many places where String.format is used unnecessarily. The test 
> builder methods already use String.format for you when you pass them args. I 
> cleaned some of these up.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236392#comment-16236392
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148628656
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
--- End diff --

It must be shared among several queries.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236393#comment-16236393
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148628710
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBDatabaseSchema.java
 ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import 
org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory.OpenTSDBTables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.Set;
+
+public class OpenTSDBDatabaseSchema extends AbstractSchema {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(OpenTSDBDatabaseSchema.class);
+
+  private final OpenTSDBTables schema;
+  private final Set tableNames;
+
+  private final Map drillTables = Maps.newHashMap();
+
+  public OpenTSDBDatabaseSchema(Set tableList, OpenTSDBTables 
schema,
+String name) {
+super(schema.getSchemaPath(), name);
+this.schema = schema;
+this.tableNames = Sets.newHashSet(tableList);
--- End diff --

I rewrited `OpenTSDBSchemaFactory`, and for that moment 
`OpenTSDBDatabaseSchema` not needed.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236391#comment-16236391
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148628583
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDB.java
 ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import retrofit2.Call;
+import retrofit2.http.Body;
+import retrofit2.http.GET;
+import retrofit2.http.POST;
+import retrofit2.http.Query;
+
+import java.util.Set;
+
+/**
+ * Client for API requests to openTSDB
+ */
+public interface OpenTSDB {
+
+  /**
+   * Used for getting all metrics names from openTSDB
+   *
+   * @return Set with all tables names
+   */
+  @GET("api/suggest?type=metrics=999")
--- End diff --

I do not find another way of getting all the metrics from openTSDB. And I 
don't know in what place we can configure this value. I suppose we can set the 
max value for this parameter and get all the metrics.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4211) Column aliases not pushed down to JDBC stores in some cases when Drill expects aliased columns to be returned.

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-4211:
-
Fix Version/s: (was: 1.12.0)

> Column aliases not pushed down to JDBC stores in some cases when Drill 
> expects aliased columns to be returned.
> --
>
> Key: DRILL-4211
> URL: https://issues.apache.org/jira/browse/DRILL-4211
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.11.0
> Environment: Postgres db storage
>Reporter: Robert Hamilton-Smith
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: newbie
>
> When making an sql statement that incorporates a join to a table and then a 
> self join to that table to get a parent value , Drill brings back 
> inconsistent results. 
> Here is the sql in postgres with correct output:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from transactions trx
> join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join categories w1 on (cat.categoryparentguid = w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL;
> {code}
> Output:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|food|
> |id1|restaurants|food|
> |id2|Coffee Shops|food|
> |id2|Coffee Shops|food|
> When run in Drill with correct storage prefix:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from db.schema.transactions trx
> join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join db.schema.wpfm_categories w1 on (cat.categoryparentguid = 
> w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL
> {code}
> Results are:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|null|
> |id1|restaurants|null|
> |id2|Coffee Shops|null|
> |id2|Coffee Shops|null|
> Physical plan is:
> {code:sql}
> 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) 
> categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = 
> {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293
> 00-01  Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) : 
> rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, 
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292
> 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47]) 
> : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname, 
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291
> 00-03  Jdbc(sql=[SELECT *
> FROM "public"."transactions"
> INNER JOIN (SELECT *
> FROM "public"."categories"
> WHERE "categoryparentguid" IS NOT NULL) AS "t" ON 
> "transactions"."categoryguid" = "t"."categoryguid"
> INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid" 
> = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255) 
> transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255) 
> transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50) 
> transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid, 
> VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate, 
> VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3) 
> transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3) 
> transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512) 
> transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20) 
> transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode, 
> VARCHAR(50) transactionorigpartyguid, VARCHAR(255) 
> transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255) 
> transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1, 
> 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0) 
> transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6) 
> transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype, VARCHAR(50) 
> transactionmatrulerefobjguid, VARCHAR(50) transactionmatrulerefobjvalue, 
> VARCHAR(50) transactionuserruleguid, DECIMAL(2, 0) transactionsplitorder, 
> TIMESTAMP(6) transactionprocessedat, TIMESTAMP(6) 
> transactioncategoryassignat, VARCHAR(50) transactionsystemcategoryguid, 
> VARCHAR(50) transactionorigmandateid, VARCHAR(100) fingerprint, VARCHAR(50) 
> categoryguid0, VARCHAR(50) categoryparentguid, DECIMAL(3, 0) categorytype, 
> VARCHAR(50) categoryname, VARCHAR(50) categorydescription, VARCHAR(50) 
> partyguid, VARCHAR(50) 

[jira] [Updated] (DRILL-5783) Make code generation in the TopN operator more modular and test it

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-5783:
-
Description: [~timothyfarkas] to add more details here

> Make code generation in the TopN operator more modular and test it
> --
>
> Key: DRILL-5783
> URL: https://issues.apache.org/jira/browse/DRILL-5783
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> [~timothyfarkas] to add more details here



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236360#comment-16236360
 ] 

ASF GitHub Bot commented on DRILL-5923:
---

GitHub user prasadns14 opened a pull request:

https://github.com/apache/drill/pull/1021

DRILL-5923: Display name for query state

Defined display names for query state. 
@paul-rogers please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/prasadns14/drill DRILL-5922

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1021


commit 39c5e18c705728a9c47b827a18f5cd44d53dde27
Author: Prasad Nagaraj Subramanya 
Date:   2017-11-02T18:36:53Z

DRILL-5923: Display name for query state




> State of a successfully completed query shown as "COMPLETED"
> 
>
> Key: DRILL-5923
> URL: https://issues.apache.org/jira/browse/DRILL-5923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Prasad Nagaraj Subramanya
>Priority: Major
> Fix For: 1.12.0
>
>
> Drill UI currently lists a successfully completed query as "COMPLETED". 
> Successfully completed, failed and canceled queries are all grouped as 
> Completed queries. 
> It would be better to list the state of a successfully completed query as 
> "Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas updated DRILL-5922:
--
Description: 
This issue was originall found by [~ben-zvi]. I am able to consistently 
reproduce the error on my laptop by running the following unit test:

org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery

{code}
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child 
allocators.
Allocator(ROOT) 0/1048576/10113536/3221225472 (res/actual/peak/limit)
  child allocators: 1
Allocator(query:26049b50-0cec-0a92-437c-bbe486e1fcbf) 1048576/0/0/268435456 
(res/actual/peak/limit)
  child allocators: 0
  ledgers: 0
  reservations: 0
  ledgers: 0
  reservations: 0

at 
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) 
~[classes/:na]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
[classes/:na]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
[classes/:na]
at 
org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256) 
~[classes/:na]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) 
[classes/:na]
at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) 
[classes/:na]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205) 
[classes/:na]
at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315) 
[test-classes/:na]
at 
org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:157) 
[test-classes/:na]
at 
org.apache.drill.BaseTestQuery.updateTestCluster(BaseTestQuery.java:148) 
[test-classes/:na]
at 
org.apache.drill.exec.DrillSeparatePlanningTest.getFragmentsHelper(DrillSeparatePlanningTest.java:185)
 [test-classes/:na]
at 
org.apache.drill.exec.DrillSeparatePlanningTest.testMultiMinorFragmentComplexQuery(DrillSeparatePlanningTest.java:108)
 [test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[na:1.8.0_144]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[na:1.8.0_144]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[na:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 [junit-4.11.jar:na]
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 [junit-4.11.jar:na]
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
 [junit-4.11.jar:na]
at 
mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
 [jmockit-1.3.jar:na]
at 
mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
 [jmockit-1.3.jar:na]
at 
mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
 [jmockit-1.3.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[na:1.8.0_144]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[na:1.8.0_144]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[na:1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
at 
mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
 [jmockit-1.3.jar:na]
at 
mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76) 
[jmockit-1.3.jar:na]
at 
mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41) 
[jmockit-1.3.jar:na]
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) 
[junit-4.11.jar:na]
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 [junit-4.11.jar:na]
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 [junit-4.11.jar:na]

{code}

> Intermittent Memory Leaks in the ROOT allocator  
> -
>
> Key: DRILL-5922
> URL: https://issues.apache.org/jira/browse/DRILL-5922
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
>
> This issue was originall found by [~ben-zvi]. I am able to consistently 
> reproduce the error on my laptop by running the following unit test:
> org.apache.drill.exec.DrillSeparatePlanningTest#testMultiMinorFragmentComplexQuery
> {code}

[jira] [Commented] (DRILL-5864) Selecting a non-existing field from a MapR-DB JSON table fails with NPE

[Pritesh Maker (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236278#comment-16236278
 ] 

Pritesh Maker commented on DRILL-5864:
--

[~HanumathRao] this PR is for this JIRA, right? 
https://github.com/apache/drill/pull/1007


> Selecting a non-existing field from a MapR-DB JSON table fails with NPE
> ---
>
> Key: DRILL-5864
> URL: https://issues.apache.org/jira/browse/DRILL-5864
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators, Storage - MapRDB
>Affects Versions: 1.12.0
>Reporter: Abhishek Girish
>Assignee: Hanumath Rao Maduri
>Priority: Major
>  Labels: ready-to-commit
> Attachments: OrderByNPE.log, OrderByNPE2.log
>
>
> Query 1
> {code}
> > select C_FIRST_NAME,C_BIRTH_COUNTRY,C_BIRTH_YEAR,C_BIRTH_MONTH,C_BIRTH_DAY 
> > from customer ORDER BY C_BIRTH_COUNTRY ASC, C_FIRST_NAME ASC LIMIT 10;
> Error: SYSTEM ERROR: NullPointerException
>   (java.lang.NullPointerException) null
> org.apache.drill.exec.record.SchemaUtil.coerceContainer():176
> 
> org.apache.drill.exec.physical.impl.xsort.managed.BufferedBatches.convertBatch():124
> org.apache.drill.exec.physical.impl.xsort.managed.BufferedBatches.add():90
> org.apache.drill.exec.physical.impl.xsort.managed.SortImpl.addBatch():265
> 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():421
> 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():357
> 
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():302
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext():115
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext():115
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():134
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624
> java.lang.Thread.run():748 (state=,code=0)
> {code}
> Plan
> {code}
> 00-00Screen
> 00-01  Project(C_FIRST_NAME=[$0], C_BIRTH_COUNTRY=[$1], 
> C_BIRTH_YEAR=[$2], C_BIRTH_MONTH=[$3], C_BIRTH_DAY=[$4])
> 00-02SelectionVectorRemover
> 00-03  Limit(fetch=[10])
> 00-04Limit(fetch=[10])
> 00-05  SelectionVectorRemover
> 00-06Sort(sort0=[$1], sort1=[$0], dir0=[ASC], dir1=[ASC])
> 00-07  Scan(groupscan=[JsonTableGroupScan 
> [ScanSpec=JsonScanSpec 
> 

[jira] [Created] (DRILL-5923) State of a successfully completed query shown as "COMPLETED"

[Prasad Nagaraj Subramanya (JIRA)]

Prasad Nagaraj Subramanya created DRILL-5923:


 Summary: State of a successfully completed query shown as 
"COMPLETED"
 Key: DRILL-5923
 URL: https://issues.apache.org/jira/browse/DRILL-5923
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - HTTP
Affects Versions: 1.11.0
Reporter: Prasad Nagaraj Subramanya
Assignee: Prasad Nagaraj Subramanya
Priority: Major
 Fix For: 1.12.0


Drill UI currently lists a successfully completed query as "COMPLETED". 
Successfully completed, failed and canceled queries are all grouped as 
Completed queries. 

It would be better to list the state of a successfully completed query as 
"Succeeded" to avoid confusion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5922) Intermittent Memory Leaks in the ROOT allocator

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-5922:
-

 Summary: Intermittent Memory Leaks in the ROOT allocator  
 Key: DRILL-5922
 URL: https://issues.apache.org/jira/browse/DRILL-5922
 Project: Apache Drill
  Issue Type: Bug
Reporter: Timothy Farkas
Assignee: Timothy Farkas
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5834) Add Networking Functions

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236129#comment-16236129
 ] 

Arina Ielchiieva commented on DRILL-5834:
-

PR-971 is outdated, please refer to PR-1018.

> Add Networking Functions
> 
>
> Key: DRILL-5834
> URL: https://issues.apache.org/jira/browse/DRILL-5834
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Affects Versions: 1.11.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Minor
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> On the heels of the PCAP plugin, this is a collection of functions that would 
> facilitate network analysis using Drill. 
> The functions include:
> inet_aton(): Converts an IPv4 address into an integer.
> inet_ntoa( ): Converts an integer IP into dotted decimal notation
> in_network( , ): Returns true if the IP address is in the given 
> CIDR block
> address_count(  ): Returns the number of IPs in a given CIDR block
> broadcast_address(  ): Returns the broadcast address for a given CIDR 
> block
> netmask( ): Returns the netmask for a given CIDR block.
> low_address(): Returns the first address in a given CIDR block.
> high_address(): Returns the last address in a given CIDR block.
> url_encode(  ): Returns a URL encoded string.
> url_decode(  ): Decodes a URL encoded string.
> is_valid_IP(): Returns true if the IP is a valid IP address
> is_private_ip(): Returns true if the IP is a private IPv4 address
> is_valid_IPv4(): Returns true if the IP is a valid IPv4 address
> is_valid_IPv6(): Returns true if the IP is a valid IPv6 address



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235965#comment-16235965
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569546
  
--- Diff: 
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestOpenTSDBPlugin.java
 ---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson;
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WTIHOUT_TAGS;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITHOUT_TAGS;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITH_TAGS;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.REQUEST_TO_NONEXISTENT_METRIC;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_NAME_REQUEST;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_REQUEST;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITHOUT_TAGS;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static 
org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS;
+
+public class TestOpenTSDBPlugin extends BaseTestQuery {
--- End diff --

This type of query unsupported in this realization.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235970#comment-16235970
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569640
  
--- Diff: exec/java-exec/src/test/resources/drill-module.conf ---
@@ -10,7 +10,7 @@ drill: {
 packages += "org.apache.drill.exec.testing",
 packages += "org.apache.drill.exec.rpc.user.security.testing"
   }
-  test.query.printing.silent : false, 
+  test.query.printing.silent : false,
--- End diff --

Ok.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235968#comment-16235968
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569604
  
--- Diff: distribution/pom.xml ---
@@ -190,11 +190,21 @@
 
 
   org.apache.drill.contrib
+  drill-opentsdb-storage
--- End diff --

This is merge mistake. I will solve this problem.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235964#comment-16235964
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569508
  
--- Diff: 
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
+
+  static final String SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS = "[{" +
--- End diff --

Sure.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235969#comment-16235969
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569621
  
--- Diff: distribution/src/assemble/bin.xml ---
@@ -99,7 +99,9 @@
 org.apache.drill.contrib:drill-format-mapr
 org.apache.drill.contrib:drill-jdbc-storage
 org.apache.drill.contrib:drill-kudu-storage
+org.apache.drill.contrib:drill-opentsdb-storage
--- End diff --

It is the same problem as in pom file. I will solve it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235966#comment-16235966
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569566
  
--- Diff: 
contrib/storage-opentsdb/src/test/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+  "storage": {
+openTSDB: {
+  type: "openTSDB",
+  connection: "localhost:8089",
--- End diff --

Sure.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235967#comment-16235967
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569587
  
--- Diff: contrib/storage-opentsdb/src/test/resources/logback.xml ---
@@ -0,0 +1,64 @@
+
--- End diff --

Ok, I will remove this file.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235961#comment-16235961
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569460
  
--- Diff: 
contrib/storage-opentsdb/src/main/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+  "storage": {
+openTSDB: {
+  type: "openTSDB",
+  connection: "1.2.3.4:4242",
+  enabled: true
--- End diff --

For now, a user cannot choose what type of connection will be used by 
drill. But it must be configurable. I will change it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235960#comment-16235960
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569425
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
+
+  public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String 
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+  }
+
+  @Override
+  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus 
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+  }
+
+  class OpenTSDBTables extends AbstractSchema {
+private final Map schemaMap = 
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+  super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+  Set tables;
+  if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, 
name));
+  }
+  return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+  return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+  OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+  name = getValidTableName(name);
+  try {
+return new DrillOpenTSDBTable(schemaName, plugin, new 
Schema(plugin.getClient(), name), scanSpec);
+  } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
+return null;
+  }
+}
+
+@Override
+public Set getTableNames() {
+  return plugin.getClient().getAllTableNames();
+}
+
+@Override
+public CreateTableEntry createNewTable(final String tableName, 
List partitionColumns) {
+  return null;
+}
+
+@Override
+public void dropTable(String tableName) {
+}
+
+@Override
+public boolean isMutable() {
--- End diff --

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235952#comment-16235952
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569297
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
+
+  public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String 
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+  }
+
+  @Override
+  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus 
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+  }
+
+  class OpenTSDBTables extends AbstractSchema {
--- End diff --

Ok, good idea.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235951#comment-16235951
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569232
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
--- End diff --

Yes, missed it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235957#comment-16235957
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569362
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
+
+  public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String 
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+  }
+
+  @Override
+  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus 
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+  }
+
+  class OpenTSDBTables extends AbstractSchema {
+private final Map schemaMap = 
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+  super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+  Set tables;
+  if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, 
name));
+  }
+  return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+  return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+  OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+  name = getValidTableName(name);
+  try {
+return new DrillOpenTSDBTable(schemaName, plugin, new 
Schema(plugin.getClient(), name), scanSpec);
+  } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
--- End diff --

I think we must throw a meaningful exception to the user. I will change it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235959#comment-16235959
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569400
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
+
+  public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String 
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+  }
+
+  @Override
+  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus 
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+  }
+
+  class OpenTSDBTables extends AbstractSchema {
+private final Map schemaMap = 
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+  super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+  Set tables;
+  if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, 
name));
+  }
+  return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+  return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+  OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+  name = getValidTableName(name);
+  try {
+return new DrillOpenTSDBTable(schemaName, plugin, new 
Schema(plugin.getClient(), name), scanSpec);
+  } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
+return null;
+  }
+}
+
+@Override
+public Set getTableNames() {
+  return plugin.getClient().getAllTableNames();
+}
+
+@Override
+public CreateTableEntry createNewTable(final String tableName, 
List partitionColumns) {
+  return null;
+}
+
+@Override
+public void dropTable(String tableName) {
--- End diff --

Ok, I will remove this overriding of the method.


> OpenTSDB storage 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235956#comment-16235956
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569335
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
+
+  public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String 
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+  }
+
+  @Override
+  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus 
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+  }
+
+  class OpenTSDBTables extends AbstractSchema {
+private final Map schemaMap = 
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+  super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+  Set tables;
+  if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, 
name));
+  }
+  return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+  return Collections.emptySet();
--- End diff --

Yes, I will remove it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235963#comment-16235963
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569487
  
--- Diff: 
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
--- End diff --

Ok.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235962#comment-16235962
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569476
  
--- Diff: 
contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestBase.java
 ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.junit.Assert;
+
+import java.util.List;
+
+class TestBase extends PlanTestBase {
--- End diff --

Ok.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235950#comment-16235950
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569173
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(ServiceImpl.class);
+
+  private OpenTSDB client;
+  private Map queryParameters;
+
+  public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+  }
+
+  @Override
+  public Set getTablesFromDB() {
+return getAllMetricsByTags();
+  }
+
+  @Override
+  public Set getAllTableNames() {
+return getTableNames();
+  }
+
+  @Override
+  public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+  for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+  unfixedColumns.add(tmp);
+}
+  }
+}
+return unfixedColumns;
+  }
+
+  @Override
+  public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+  this.queryParameters = parseFROMRowData(rowData);
+} else {
+  Map params = new HashMap<>();
+  params.put(METRIC, rowData);
+  this.queryParameters = params;
+}
+  }
+
+  private Set getAllMetricsByTags() {
+try {
+  return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+  logIOException(e);
+  return Collections.emptySet();
+}
+  }
+
+  private Set getTableNames() {
+try {
+  return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+  e.printStackTrace();
+  return Collections.emptySet();
+}
+  }
+
+  private Set getAllTablesWithSpecialTag(DBQuery base) throws 
IOException {
+return 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235958#comment-16235958
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569383
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+  private final String schemaName;
+  private OpenTSDBStoragePlugin plugin;
+
+  public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String 
schemaName) {
+this.plugin = plugin;
+this.schemaName = schemaName;
+  }
+
+  @Override
+  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus 
parent) throws IOException {
+OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+parent.add(schemaName, schema);
+  }
+
+  class OpenTSDBTables extends AbstractSchema {
+private final Map schemaMap = 
Maps.newHashMap();
+
+OpenTSDBTables(String name) {
+  super(Collections.emptyList(), name);
+}
+
+@Override
+public AbstractSchema getSubSchema(String name) {
+  Set tables;
+  if (!schemaMap.containsKey(name)) {
+tables = plugin.getClient().getAllTableNames();
+schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, 
name));
+  }
+  return schemaMap.get(name);
+}
+
+@Override
+public Set getSubSchemaNames() {
+  return Collections.emptySet();
+}
+
+@Override
+public Table getTable(String name) {
+  OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+  name = getValidTableName(name);
+  try {
+return new DrillOpenTSDBTable(schemaName, plugin, new 
Schema(plugin.getClient(), name), scanSpec);
+  } catch (Exception e) {
+log.warn("Failure while retrieving openTSDB table {}", name, e);
+return null;
+  }
+}
+
+@Override
+public Set getTableNames() {
+  return plugin.getClient().getAllTableNames();
+}
+
+@Override
+public CreateTableEntry createNewTable(final String tableName, 
List partitionColumns) {
+  return null;
--- End diff --

Ok, I will remove this overriding of the method.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235940#comment-16235940
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568871
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java
 ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static 
org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ * 
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+  /**
+   * The name of an aggregation function to use.
+   */
+  private String aggregator;
+  /**
+   * The name of a metric stored in the system
+   */
+  private String metric;
+  /**
+   * Whether or not the data should be converted into deltas before 
returning.
+   * This is useful if the metric is a continuously incrementing counter
+   * and you want to view the rate of change between data points.
+   */
+  private String rate;
+  /**
+   * An optional downsampling function to reduce the amount of data 
returned.
+   */
+  private String downsample;
+  /**
+   * To drill down to specific timeseries or group results by tag,
+   * supply one or more map values in the same format as the query string.
+   */
+  private Map tags;
+
+  private Query(Builder builder) {
+this.aggregator = builder.aggregator;
+this.metric = builder.metric;
+this.rate = builder.rate;
+this.downsample = builder.downsample;
+this.tags = builder.tags;
+  }
+
+  public String getAggregator() {
+return aggregator;
+  }
+
+  public String getMetric() {
+return metric;
+  }
+
+  public String getRate() {
+return rate;
+  }
+
+  public String getDownsample() {
+return downsample;
+  }
+
+  public Map getTags() {
+return tags;
+  }
+
+  public static class Builder {
+
+private String aggregator = SUM_AGGREGATOR;
--- End diff --

Sure, I will remove support for these type of queries.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235944#comment-16235944
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568941
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(ServiceImpl.class);
+
+  private OpenTSDB client;
+  private Map queryParameters;
+
+  public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+  }
+
+  @Override
+  public Set getTablesFromDB() {
--- End diff --

Sure, I will add comments for methods in the Service interface.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235948#comment-16235948
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148569087
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(ServiceImpl.class);
+
+  private OpenTSDB client;
+  private Map queryParameters;
+
+  public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+  }
+
+  @Override
+  public Set getTablesFromDB() {
+return getAllMetricsByTags();
+  }
+
+  @Override
+  public Set getAllTableNames() {
+return getTableNames();
+  }
+
+  @Override
+  public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+  for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+  unfixedColumns.add(tmp);
+}
+  }
+}
+return unfixedColumns;
+  }
+
+  @Override
+  public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+  this.queryParameters = parseFROMRowData(rowData);
+} else {
+  Map params = new HashMap<>();
+  params.put(METRIC, rowData);
+  this.queryParameters = params;
+}
+  }
+
+  private Set getAllMetricsByTags() {
+try {
+  return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+  logIOException(e);
+  return Collections.emptySet();
+}
+  }
+
+  private Set getTableNames() {
+try {
+  return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+  e.printStackTrace();
+  return Collections.emptySet();
+}
+  }
+
+  private Set getAllTablesWithSpecialTag(DBQuery base) throws 
IOException {
+return 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235941#comment-16235941
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568893
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java
 ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static 
org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ * 
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+  /**
+   * The name of an aggregation function to use.
+   */
+  private String aggregator;
+  /**
+   * The name of a metric stored in the system
+   */
+  private String metric;
+  /**
+   * Whether or not the data should be converted into deltas before 
returning.
+   * This is useful if the metric is a continuously incrementing counter
+   * and you want to view the rate of change between data points.
+   */
+  private String rate;
+  /**
+   * An optional downsampling function to reduce the amount of data 
returned.
+   */
+  private String downsample;
+  /**
+   * To drill down to specific timeseries or group results by tag,
+   * supply one or more map values in the same format as the query string.
+   */
+  private Map tags;
+
+  private Query(Builder builder) {
+this.aggregator = builder.aggregator;
+this.metric = builder.metric;
+this.rate = builder.rate;
+this.downsample = builder.downsample;
+this.tags = builder.tags;
+  }
+
+  public String getAggregator() {
+return aggregator;
+  }
+
+  public String getMetric() {
+return metric;
+  }
+
+  public String getRate() {
+return rate;
+  }
+
+  public String getDownsample() {
+return downsample;
+  }
+
+  public Map getTags() {
+return tags;
+  }
+
+  public static class Builder {
+
+private String aggregator = SUM_AGGREGATOR;
+private String metric;
+private String rate;
+private String downsample;
+private Map tags = new HashMap<>();
+
+public Builder(String metric) {
+  this.metric = metric;
+}
+
+public Builder setAggregator(String aggregator) {
+  if (aggregator != null) {
+this.aggregator = aggregator;
+  }
+  return this;
+}
+
+public Builder setMetric(String metric) {
+  this.metric = metric;
+  return this;
+}
+
+public Builder setRate(String rate) {
+  if (rate != null) {
--- End diff --

Ok, I will remove it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235937#comment-16235937
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568797
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDBTypes.java
 ---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+/**
+ * Types in openTSDB records,
+ * used for converting openTSDB data to Sql representation
+ */
+public enum OpenTSDBTypes {
+  STRING,
+  DOUBLE,
+  TIMESTAMP,
--- End diff --

Ok.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235939#comment-16235939
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568843
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java
 ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request 
to DB.
+ * 
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result 
set.
+ */
+public class DBQuery {
+
+  /**
+   * The start time for the query. This can be a relative or absolute 
timestamp.
+   */
+  private String start;
+  /**
+   * One or more sub subQueries used to select the time series to return.
+   */
+  private Set queries;
+
+  private DBQuery(Builder builder) {
+this.start = builder.start;
+this.queries = builder.queries;
+  }
+
+  public String getStart() {
+return start;
+  }
+
+  public Set getQueries() {
+return queries;
+  }
+
+  public static class Builder {
+
+private String start = DEFAULT_TIME;
+private Set queries = new HashSet<>();
--- End diff --

Yes, good idea, I will add this type of check to the `Builder`.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235945#comment-16235945
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568965
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(ServiceImpl.class);
+
+  private OpenTSDB client;
+  private Map queryParameters;
+
+  public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+  }
+
+  @Override
+  public Set getTablesFromDB() {
+return getAllMetricsByTags();
+  }
+
+  @Override
+  public Set getAllTableNames() {
+return getTableNames();
+  }
+
+  @Override
+  public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+  for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+  unfixedColumns.add(tmp);
+}
+  }
+}
+return unfixedColumns;
+  }
+
+  @Override
+  public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+  this.queryParameters = parseFROMRowData(rowData);
+} else {
+  Map params = new HashMap<>();
+  params.put(METRIC, rowData);
+  this.queryParameters = params;
+}
+  }
+
+  private Set getAllMetricsByTags() {
+try {
+  return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+  logIOException(e);
--- End diff --

No, it is the critical exception. I will add exception throwing with the 
detailed message.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>   

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235938#comment-16235938
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568822
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java
 ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request 
to DB.
+ * 
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result 
set.
+ */
+public class DBQuery {
+
+  /**
+   * The start time for the query. This can be a relative or absolute 
timestamp.
+   */
+  private String start;
+  /**
+   * One or more sub subQueries used to select the time series to return.
+   */
+  private Set queries;
+
+  private DBQuery(Builder builder) {
+this.start = builder.start;
+this.queries = builder.queries;
+  }
+
+  public String getStart() {
+return start;
+  }
+
+  public Set getQueries() {
+return queries;
+  }
+
+  public static class Builder {
+
+private String start = DEFAULT_TIME;
--- End diff --

Sure, I will change this behavior, and this parameter will be required for 
all select queries.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235942#comment-16235942
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568915
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(ServiceImpl.class);
+
+  private OpenTSDB client;
--- End diff --

Sure, it must be final.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235946#comment-16235946
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568996
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java
 ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(ServiceImpl.class);
+
+  private OpenTSDB client;
+  private Map queryParameters;
+
+  public ServiceImpl(String connectionURL) {
+this.client = new Retrofit.Builder()
+.baseUrl(connectionURL)
+.addConverterFactory(JacksonConverterFactory.create())
+.build()
+.create(OpenTSDB.class);
+  }
+
+  @Override
+  public Set getTablesFromDB() {
+return getAllMetricsByTags();
+  }
+
+  @Override
+  public Set getAllTableNames() {
+return getTableNames();
+  }
+
+  @Override
+  public List getUnfixedColumnsToSchema() {
+Set tables = getAllMetricsByTags();
+List unfixedColumns = new ArrayList<>();
+
+for (MetricDTO table : tables) {
+  for (String tag : table.getTags().keySet()) {
+ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+if (!unfixedColumns.contains(tmp)) {
+  unfixedColumns.add(tmp);
+}
+  }
+}
+return unfixedColumns;
+  }
+
+  @Override
+  public void setupQueryParameters(String rowData) {
+if (!isTableNameValid(rowData)) {
+  this.queryParameters = parseFROMRowData(rowData);
+} else {
+  Map params = new HashMap<>();
+  params.put(METRIC, rowData);
+  this.queryParameters = params;
+}
+  }
+
+  private Set getAllMetricsByTags() {
+try {
+  return getAllMetricsFromDBByTags();
+} catch (IOException e) {
+  logIOException(e);
+  return Collections.emptySet();
+}
+  }
+
+  private Set getTableNames() {
+try {
+  return client.getAllTablesName().execute().body();
+} catch (IOException e) {
+  e.printStackTrace();
+  return Collections.emptySet();
--- End diff --

This is really bad practice, sorry for that, I will change it.


> OpenTSDB storage plugin
> ---
>
>

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235936#comment-16235936
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568766
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java
 ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+  /**
+   * Parse FROM parameters to Map representation
+   *
+   * @param rowData with this syntax (metric=warp.speed.test)
+   * @return Map with params key: metric, value: warp.speed.test
+   */
+  public static Map parseFROMRowData(String rowData) {
--- End diff --

I will change it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235934#comment-16235934
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568739
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java
 ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+  @JsonProperty
+  public final OpenTSDBStoragePluginConfig storage;
+
+  private final List columns;
+  private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+  private final List tabletScanSpecList;
+
+  @JsonCreator
+  public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig 
storage,
+ @JsonProperty("tabletScanSpecList") 
LinkedList tabletScanSpecList,
+ @JsonProperty("columns") List 
columns) throws ExecutionSetupException {
+super((String) null);
+openTSDBStoragePlugin = (OpenTSDBStoragePlugin) 
registry.getPlugin(storage);
+this.tabletScanSpecList = tabletScanSpecList;
+this.storage = (OpenTSDBStoragePluginConfig) storage;
+this.columns = columns;
+  }
+
+  public OpenTSDBSubScan(OpenTSDBStoragePlugin plugin, 
OpenTSDBStoragePluginConfig config,
+ List tabletInfoList, 
List columns) {
+super((String) null);
+openTSDBStoragePlugin = plugin;
+storage = config;
+this.tabletScanSpecList = tabletInfoList;
+this.columns = columns;
+  }
+
+  @Override
+  public int getOperatorType() {
+return 0;
+  }
+
+  @Override
+  public boolean isExecutable() {
+return false;
+  }
+
+  @Override
+  public PhysicalOperator getNewWithChildren(List 
children) throws ExecutionSetupException {
+Preconditions.checkArgument(children.isEmpty());
+return new OpenTSDBSubScan(openTSDBStoragePlugin, storage, 
tabletScanSpecList, columns);
+  }
+
+  @Override
+  public Iterator iterator() {
+return Collections.emptyIterator();
+  }
+
+  @Override
+  public  T accept(PhysicalVisitor 
physicalVisitor, X value) throws E {
+return physicalVisitor.visitSubScan(this, value);
+  }
+
+  public List getColumns() {
+return columns;
+  }
+
+  public List getTabletScanSpecList() {
+return tabletScanSpecList;
+  }
+
+  @JsonIgnore

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235925#comment-16235925
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568092
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java
 ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+  private static final Logger log =
+  LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+  @JsonProperty
+  public final OpenTSDBStoragePluginConfig storage;
+
+  private final List columns;
+  private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+  private final List tabletScanSpecList;
+
+  @JsonCreator
+  public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig 
storage,
+ @JsonProperty("tabletScanSpecList") 
LinkedList tabletScanSpecList,
+ @JsonProperty("columns") List 
columns) throws ExecutionSetupException {
+super((String) null);
+openTSDBStoragePlugin = (OpenTSDBStoragePlugin) 
registry.getPlugin(storage);
+this.tabletScanSpecList = tabletScanSpecList;
+this.storage = (OpenTSDBStoragePluginConfig) storage;
--- End diff --

Thanks. I will change this casting.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235919#comment-16235919
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567893
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
 ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+  private static final Map TYPES;
+
+  private Service db;
+
+  private Iterator tableIterator;
+  private OutputMutator output;
+  private ImmutableList projectedCols;
+  private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+  OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec 
subScanSpec,
+   List projectedColumns) throws 
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+  }
+
+  @Override
+  public void setup(OperatorContext context, OutputMutator output) throws 
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+  throw new ValidationError(String.format("Table '%s' not found or 
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+  }
+
+  @Override
+  public int next() {
+try {
+  return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+  log.info(e.toString());
+  return 0;
+}
+  }
+
+  @Override
+  protected boolean isSkipQuery() {
+return super.isSkipQuery();
+  }
+
+  @Override
+  

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235922#comment-16235922
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568016
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+  private final DrillbitContext context;
+
+  private final OpenTSDBStoragePluginConfig engineConfig;
+  private final OpenTSDBSchemaFactory schemaFactory;
+
+  private final ServiceImpl db;
+
+  public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, 
DrillbitContext context, String name) throws IOException {
+this.context = context;
+this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+this.engineConfig = configuration;
+this.db = new ServiceImpl("http://; + configuration.getConnection());
+  }
+
+  @Override
+  public void start() throws IOException {
--- End diff --

Sure, I will remove it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235915#comment-16235915
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567810
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
 ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+  private static final Map TYPES;
+
+  private Service db;
+
+  private Iterator tableIterator;
+  private OutputMutator output;
+  private ImmutableList projectedCols;
+  private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+  OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec 
subScanSpec,
+   List projectedColumns) throws 
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+  }
+
+  @Override
+  public void setup(OperatorContext context, OutputMutator output) throws 
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+  throw new ValidationError(String.format("Table '%s' not found or 
it's empty", subScanSpec.getTableName()));
--- End diff --

Ok, this is really weird behaviour, I will change it. Yes, 
`db.getTablesFromDB();` returns all the metrics from db. I will add comments to 
the service methods. 
Also, I completely agree that the exception handling in this place really 
confusing, I will change it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235921#comment-16235921
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567959
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+  private final DrillbitContext context;
+
+  private final OpenTSDBStoragePluginConfig engineConfig;
+  private final OpenTSDBSchemaFactory schemaFactory;
+
+  private final ServiceImpl db;
+
+  public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, 
DrillbitContext context, String name) throws IOException {
+this.context = context;
+this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+this.engineConfig = configuration;
+this.db = new ServiceImpl("http://; + configuration.getConnection());
--- End diff --

Thank for this remark, I will move this choice to the config's.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235916#comment-16235916
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567835
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
 ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+  private static final Map TYPES;
+
+  private Service db;
+
+  private Iterator tableIterator;
+  private OutputMutator output;
+  private ImmutableList projectedCols;
+  private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+  OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec 
subScanSpec,
+   List projectedColumns) throws 
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+  }
+
+  @Override
+  public void setup(OperatorContext context, OutputMutator output) throws 
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+  throw new ValidationError(String.format("Table '%s' not found or 
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+  }
+
+  @Override
+  public int next() {
+try {
+  return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+  log.info(e.toString());
--- End diff --

Sure, I will replace it. 


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235923#comment-16235923
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568031
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+  private final DrillbitContext context;
+
+  private final OpenTSDBStoragePluginConfig engineConfig;
+  private final OpenTSDBSchemaFactory schemaFactory;
+
+  private final ServiceImpl db;
+
+  public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, 
DrillbitContext context, String name) throws IOException {
+this.context = context;
+this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+this.engineConfig = configuration;
+this.db = new ServiceImpl("http://; + configuration.getConnection());
+  }
+
+  @Override
+  public void start() throws IOException {
+  }
+
+  @Override
+  public void close() throws Exception {
+  }
+
+  @Override
+  public boolean supportsRead() {
+return true;
+  }
+
+  @Override
+  public OpenTSDBStoragePluginConfig getConfig() {
+return engineConfig;
+  }
+
+  @Override
+  public OpenTSDBGroupScan getPhysicalScan(String userName, JSONOptions 
selection) throws IOException {
+OpenTSDBScanSpec scanSpec = selection.getListWith(new ObjectMapper(), 
new TypeReference() {
+});
+return new OpenTSDBGroupScan(this, scanSpec, null);
+  }
+
+  @Override
+  public boolean supportsWrite() {
+return true;
--- End diff --

Ok, I will remove it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235924#comment-16235924
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148568059
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePluginConfig.java
 ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.logical.StoragePluginConfigBase;
+
+import java.io.IOException;
+import java.util.Objects;
+
+@JsonTypeName(OpenTSDBStoragePluginConfig.NAME)
+public class OpenTSDBStoragePluginConfig extends StoragePluginConfigBase {
+
+  public static final String NAME = "openTSDB";
+
+  private final String connection;
+
+  @JsonCreator
+  public OpenTSDBStoragePluginConfig(@JsonProperty("connection") String 
connection) throws IOException {
+this.connection = connection;
--- End diff --

Yes, we should. I will add null check.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235918#comment-16235918
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567870
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
 ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+  private static final Map TYPES;
+
+  private Service db;
+
+  private Iterator tableIterator;
+  private OutputMutator output;
+  private ImmutableList projectedCols;
+  private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+  OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec 
subScanSpec,
+   List projectedColumns) throws 
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+  }
+
+  @Override
+  public void setup(OperatorContext context, OutputMutator output) throws 
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+  throw new ValidationError(String.format("Table '%s' not found or 
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+  }
+
+  @Override
+  public int next() {
+try {
+  return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+  log.info(e.toString());
+  return 0;
+}
+  }
+
+  @Override
+  protected boolean isSkipQuery() {
+return super.isSkipQuery();
+  }
+
+  @Override
+  

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235917#comment-16235917
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567854
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
 ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+  private static final Map TYPES;
+
+  private Service db;
+
+  private Iterator tableIterator;
+  private OutputMutator output;
+  private ImmutableList projectedCols;
+  private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+  OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec 
subScanSpec,
+   List projectedColumns) throws 
IOException {
+setColumns(projectedColumns);
+this.db = client;
+this.subScanSpec = subScanSpec;
+db.setupQueryParameters(subScanSpec.getTableName());
+log.debug("Scan spec: {}", subScanSpec);
+  }
+
+  @Override
+  public void setup(OperatorContext context, OutputMutator output) throws 
ExecutionSetupException {
+this.output = output;
+Set tables = db.getTablesFromDB();
+if (tables == null || tables.isEmpty()) {
+  throw new ValidationError(String.format("Table '%s' not found or 
it's empty", subScanSpec.getTableName()));
+}
+this.tableIterator = tables.iterator();
+  }
+
+  @Override
+  public int next() {
+try {
+  return processOpenTSDBTablesData();
+} catch (SchemaChangeException e) {
+  log.info(e.toString());
+  return 0;
+}
+  }
+
+  @Override
+  protected boolean isSkipQuery() {
--- End diff --

I will remove it.


> OpenTSDB storage plugin
> 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235920#comment-16235920
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567934
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBScanSpec.java
 ---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class OpenTSDBScanSpec {
+
+  private final String tableName;
+
+  @JsonCreator
+  public OpenTSDBScanSpec(@JsonProperty("tableName") String tableName) {
+this.tableName = tableName;
+  }
+
+  public String getTableName() {
+return tableName;
+  }
--- End diff --

Sure, I will add `toString()`


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235912#comment-16235912
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567715
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
 ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import 
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+  private static final long DEFAULT_TABLET_SIZE = 1000;
+
+  private OpenTSDBStoragePluginConfig storagePluginConfig;
+  private OpenTSDBScanSpec openTSDBScanSpec;
+  private OpenTSDBStoragePlugin storagePlugin;
+
+  private ListMultimap assignments;
+  private List columns;
+  private List openTSDBWorkList = Lists.newArrayList();
+  private List affinities;
+
+  private boolean filterPushedDown = false;
+
+  @JsonCreator
+  public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") 
OpenTSDBScanSpec openTSDBScanSpec,
+   @JsonProperty("storage") 
OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+   @JsonProperty("columns") List 
columns,
+   @JacksonInject StoragePluginRegistry 
pluginRegistry) throws IOException, ExecutionSetupException {
+this((OpenTSDBStoragePlugin) 
pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec, 
columns);
+  }
+
+  public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+   OpenTSDBScanSpec scanSpec, List 
columns) {
+super((String) null);
+this.storagePlugin = storagePlugin;
+this.storagePluginConfig = storagePlugin.getConfig();
+this.openTSDBScanSpec = scanSpec;
+this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS : 
columns;
+init();
+  }
+
+  /**
+   * Private constructor, used for cloning.
+   *
+   * @param that The OpenTSDBGroupScan to clone
+   */
+  private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+super((String) null);
+this.columns = that.columns;

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235911#comment-16235911
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567693
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
 ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import 
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+  private static final long DEFAULT_TABLET_SIZE = 1000;
+
+  private OpenTSDBStoragePluginConfig storagePluginConfig;
+  private OpenTSDBScanSpec openTSDBScanSpec;
+  private OpenTSDBStoragePlugin storagePlugin;
+
+  private ListMultimap assignments;
+  private List columns;
+  private List openTSDBWorkList = Lists.newArrayList();
+  private List affinities;
+
+  private boolean filterPushedDown = false;
+
+  @JsonCreator
+  public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") 
OpenTSDBScanSpec openTSDBScanSpec,
+   @JsonProperty("storage") 
OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+   @JsonProperty("columns") List 
columns,
+   @JacksonInject StoragePluginRegistry 
pluginRegistry) throws IOException, ExecutionSetupException {
+this((OpenTSDBStoragePlugin) 
pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec, 
columns);
+  }
+
+  public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+   OpenTSDBScanSpec scanSpec, List 
columns) {
+super((String) null);
+this.storagePlugin = storagePlugin;
+this.storagePluginConfig = storagePlugin.getConfig();
+this.openTSDBScanSpec = scanSpec;
+this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS : 
columns;
+init();
+  }
+
+  /**
+   * Private constructor, used for cloning.
+   *
+   * @param that The OpenTSDBGroupScan to clone
+   */
+  private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+super((String) null);
+this.columns = that.columns;

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235905#comment-16235905
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567569
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java
 ---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public class Constants {
--- End diff --

Ok, I will change it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235910#comment-16235910
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567658
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
 ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import 
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+  private static final long DEFAULT_TABLET_SIZE = 1000;
+
+  private OpenTSDBStoragePluginConfig storagePluginConfig;
+  private OpenTSDBScanSpec openTSDBScanSpec;
+  private OpenTSDBStoragePlugin storagePlugin;
+
+  private ListMultimap assignments;
+  private List columns;
+  private List openTSDBWorkList = Lists.newArrayList();
+  private List affinities;
+
+  private boolean filterPushedDown = false;
+
+  @JsonCreator
+  public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") 
OpenTSDBScanSpec openTSDBScanSpec,
+   @JsonProperty("storage") 
OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+   @JsonProperty("columns") List 
columns,
--- End diff --

Thank for this advice, I will add getters.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235908#comment-16235908
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567620
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java
 ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import 
org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+  private static final long DEFAULT_TABLET_SIZE = 1000;
+
+  private OpenTSDBStoragePluginConfig storagePluginConfig;
+  private OpenTSDBScanSpec openTSDBScanSpec;
+  private OpenTSDBStoragePlugin storagePlugin;
+
+  private ListMultimap assignments;
+  private List columns;
+  private List openTSDBWorkList = Lists.newArrayList();
+  private List affinities;
+
+  private boolean filterPushedDown = false;
--- End diff --

No, this variable is not required, I will delete it.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235913#comment-16235913
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567743
  
--- Diff: 
contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java
 ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+  private static final Logger log = 
LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+  private static final Map TYPES;
+
+  private Service db;
+
+  private Iterator tableIterator;
+  private OutputMutator output;
+  private ImmutableList projectedCols;
+  private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+  OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec 
subScanSpec,
--- End diff --

Ok, I will add access modifier.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> 

[jira] [Commented] (DRILL-5337) OpenTSDB storage plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235903#comment-16235903
 ] 

ASF GitHub Bot commented on DRILL-5337:
---

Github user Vlad-Storona commented on a diff in the pull request:

https://github.com/apache/drill/pull/774#discussion_r148567538
  
--- Diff: contrib/storage-opentsdb/README.md ---
@@ -0,0 +1 @@
+# drill-storage-openTSDB
--- End diff --

Sure, I will update README with more detailed info.


> OpenTSDB storage plugin
> ---
>
> Key: DRILL-5337
> URL: https://issues.apache.org/jira/browse/DRILL-5337
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Dmitriy Gavrilovych
>Assignee: Dmitriy Gavrilovych
>Priority: Major
>  Labels: features
> Fix For: 1.12.0
>
>
> Storage plugin for OpenTSDB
> The plugin uses REST API to work with TSDB. 
> Expected queries are listed below:
> SELECT * FROM openTSDB.`warp.speed.test`;
> Return all elements from warp.speed.test table with default aggregator SUM
> SELECT * FROM openTSDB.`(metric=warp.speed.test)`;
> Return all elements from (metric=warp.speed.test) table as a previous query, 
> but with alternative FROM syntax
> SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg)`;
> Return all elements from warp.speed.test table, but with the custom aggregator
> SELECT `timestamp`, sum(`aggregated value`) FROM 
> openTSDB.`(metric=warp.speed.test, aggregator=avg)` GROUP BY `timestamp`;
> Return aggregated and grouped value by standard drill functions from 
> warp.speed.test table, but with the custom aggregator
> SELECT * FROM openTSDB.`(metric=warp.speed.test, downsample=5m-avg)`
> Return data limited by downsample



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5106) Refactor SkipRecordsInspector to exclude check for predefined file formats

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5106:
---

Assignee: Arina Ielchiieva

> Refactor SkipRecordsInspector to exclude check for predefined file formats
> --
>
> Key: DRILL-5106
> URL: https://issues.apache.org/jira/browse/DRILL-5106
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Hive
>Affects Versions: 1.9.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> After changes introduced in DRILL-4982, SkipRecordInspector is used only for 
> predefined formats (using hasHeaderFooter: false / true). But 
> SkipRecordInspector has its own check for formats where skip strategy can be 
> applied. Acceptable file formats are stored in private final Set 
> fileFormats and initialized in constructor, currently it contains only one 
> format - TextInputFormat. Now this check is redundant and may lead to 
> ignoring hasHeaderFooter setting to true for any other format except of Text.
> To do:
> 1. remove private final Set fileFormats
> 2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:
> {code}
>  if 
> (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT)))
>  {
> return propertyIntValue;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)