[GitHub] drill pull request #1203: DRILL-6289: Cluster view should show more relevant...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r180296333
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java 
---
@@ -102,6 +105,7 @@ public DrillbitEndpoint start() throws 
DrillbitStartupException, UnknownHostExce
 DrillbitEndpoint partialEndpoint = DrillbitEndpoint.newBuilder()
 .setAddress(hostName)
 .setUserPort(userPort)
+.setHttpPort(httpPort)
--- End diff --

That's tricky. The webserver doesn't come up until after the Endpoint is 
created and ready. One possible solution would be to perform the port hunt 
before the endpoint, and assign it to a newly instantiated webserver after the 
Endpoint is created.


---

Unit Testing Docs and [Discuss] Dev Docs location

[Timothy Farkas]

Hi All,

I am wrapping up a unit testing doc here 
https://github.com/ilooner/drill/blob/DRILL-6249/docs/DevDocs.md which expands 
on the unit testing presentation here 
https://docs.google.com/presentation/d/1YwURHUQRwPYH6d9AU6sF-O_shOKMraMG6KZnRy0ePaM/edit?usp=sharing
 .

I'd also like to propose adding the unit testing documentation along with other 
developer docs to the Drill master repo itself as markdown documents in the 
./docs folder. There are several

advantages to doing this:


  *   The markdown docs are rendered nicely on github and in your IDE through 
the use of a markdown plugin.
  *   You can link directly to source files inside a markdown doc with this 
strategy.
  *   The developer documentation is versioned and updated with the code.
  *   Developer's only have to deal with one repo for both code and 
documentation changes.


Other projects like Apache Apex, Apache Spark, Apache Kudu and others also 
utilize the markdown docs folder strategy. See here:

  *   https://github.com/apache/apex-malhar/tree/master/docs
  *   https://github.com/apache/spark/tree/master/docs
  *    
https://github.com/apache/kudu/tree/master/docs

Thanks,
Tim

[GitHub] drill pull request #1202: DRILL-6311: No logging information in drillbit.log...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1202#discussion_r180275891
  
--- Diff: exec/vector/pom.xml ---
@@ -69,6 +69,7 @@
   org.apache.drill
   drill-common
   ${project.version}
+  test
--- End diff --

OK, I see. @sohami I'd suggest moving the dependency to the parent exec 
module pom.


---

[jira] [Created] (DRILL-6317) Allow custom generators in the mock-scan operator. And remove generateTestData method from ValueVector.Mutators

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-6317:
-

 Summary: Allow custom generators in the mock-scan operator. And 
remove generateTestData method from ValueVector.Mutators
 Key: DRILL-6317
 URL: https://issues.apache.org/jira/browse/DRILL-6317
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Timothy Farkas
Assignee: Timothy Farkas


Currently when using the mock-scan operator, the generateTestData method of the 
ValueVector Mutator is used. This is a hardcoded test data generation method. I 
am improving mock-scan to use the FieldGen classes like mock-sub-scan does. 
This would render the generateTestData methods obsolete so we could remove them 
from the ValueVector Mutators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] drill pull request #1203: DRILL-6289: Cluster view should show more relevant...

[dvjyothsna]

Github user dvjyothsna commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r180268114
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java 
---
@@ -102,6 +105,7 @@ public DrillbitEndpoint start() throws 
DrillbitStartupException, UnknownHostExce
 DrillbitEndpoint partialEndpoint = DrillbitEndpoint.newBuilder()
 .setAddress(hostName)
 .setUserPort(userPort)
+.setHttpPort(httpPort)
--- End diff --

Recently port-hunting capability is added to WebServer. So I think we 
should get port number from the WebServer.


---

[jira] [Created] (DRILL-6316) drill-embedded trying to use external IP address instead of 127.0.0.1

[Steven Buehler (JIRA)]

Steven Buehler created DRILL-6316:
-

 Summary: drill-embedded trying to use external IP address instead 
of 127.0.0.1
 Key: DRILL-6316
 URL: https://issues.apache.org/jira/browse/DRILL-6316
 Project: Apache Drill
  Issue Type: Bug
  Components:  Server
Affects Versions: 1.13.0
Reporter: Steven Buehler


Running `bin/drill-embedded` off a MacOS High Sierra command line, instead of 
opening to localhost or 127.0.0.1, it tries connecting to my network's 
_external_ address instead.


{code:java}
Apr 09, 2018 6:45:08 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
Error: Failure in connecting to Drill: org.apache.drill.exec.rpc.RpcException: 
CONNECTION : io.netty.channel.AbstractChannel$AnnotatedConnectException: 
Connection refused: /97.106.218.141:31010 (state=,code=0)
java.sql.SQLNonTransientConnectionException: Failure in connecting to Drill: 
org.apache.drill.exec.rpc.RpcException: CONNECTION : 
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 
/97.106.218.141:31010
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:179)
at 
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
at 
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
at 
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)
at sqlline.Commands.connect(Commands.java:1083)
at sqlline.Commands.connect(Commands.java:1015)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:742)
at sqlline.SqlLine.initArgs(SqlLine.java:528)
at sqlline.SqlLine.begin(SqlLine.java:596)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: org.apache.drill.exec.rpc.RpcException: CONNECTION : 
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 
/97.106.218.141:31010
at 
org.apache.drill.exec.rpc.user.UserClient$2.connectionFailed(UserClient.java:315)
at 
org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler.connectionFailed(QueryResultHandler.java:389)
at 
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:119)
at 
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:77)
at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at 
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at 
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at 
io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
at 
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:278)
at 
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:294)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: 
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 
/97.106.218.141:31010
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
at 
org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:90)
... 14 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: 
Connection refused: /97.106.218.141:31010
at 

[GitHub] drill pull request #1202: DRILL-6311: No logging information in drillbit.log...

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1202#discussion_r180252832
  
--- Diff: exec/vector/pom.xml ---
@@ -69,6 +69,7 @@
   org.apache.drill
   drill-common
   ${project.version}
+  test
--- End diff --

@vrozov As we discussed offline drill-common-tests is not a dependency of 
the vector project itself, it is a dependency required by a maven invocation 
when a category is specified. For example in `mvn test 
-DexcludedGroups="org.apache.drill.categories.SlowTest,org.apache.drill.categories.UnlikelyTest,org.apache.drill.categories.SecurityTest"`
 the category classes are required by maven. However when `mvn test` is used 
the category classes are not required by maven.


---

[GitHub] drill pull request #1202: DRILL-6311: No logging information in drillbit.log...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1202#discussion_r180249547
  
--- Diff: exec/vector/pom.xml ---
@@ -69,6 +69,7 @@
   org.apache.drill
   drill-common
   ${project.version}
+  test
--- End diff --

I mean that there are no unit tests that are annotated with 
`@Category(SlowTest.class)` or any other `@Category`, so it is not clear why 
the dependency is necessary. 


---

[GitHub] drill pull request #1203: DRILL-6289: Cluster view should show more relevant...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r180248133
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -289,14 +314,94 @@
   });
   }
   }
-  
+
+  function remoteShutdown(button,host) {
+  var url = location.protocol + "//" + host + 
"/gracefulShutdown";
+  var result = $.ajax({
+type: 'POST',
+url: url,
+contentType : 'text/plain',
+complete: function(data) {
+alert(data.responseJSON["response"]);
+button.prop('disabled',true).css('opacity',0.5);
+}
+  });
+  }
+  
   function getRequestUrl(requestPath) {
 var protocol = location.protocol;
 var host = location.host;
 var url = protocol + "//" + host + requestPath;
 return url;
   }
-
+
+  //Iterates through all the nodes for update
+  function reloadMetrics() {
+  for (i = 1; i <= size; i++) {
+  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+  updateMetricsHtml(address, httpPort, i);
+  }
+  }
+
+  //Update memory
+  //TODO: HTTPS?
+  function updateMetricsHtml(drillbit,webport,idx) {
+var result = $.ajax({
+  type: 'GET',
--- End diff --

Makes sense. I'll set the default values to Unknown rather than zero. 


---

[GitHub] drill pull request #1202: DRILL-6311: No logging information in drillbit.log...

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1202#discussion_r180247017
  
--- Diff: exec/vector/pom.xml ---
@@ -69,6 +69,7 @@
   org.apache.drill
   drill-common
   ${project.version}
+  test
--- End diff --

The `tests` tag includes the test classes in 
drill-common. The test category classes are included in 
common/src/test/java/org/apache/drill/categories. So test category classes will 
be accessible to maven. However, looks like I forgot the test scope so the test 
classes and resources from drill-common were being included in Drill's standard 
(non-test) classpath.


---

[GitHub] drill pull request #1203: DRILL-6289: Cluster view should show more relevant...

[dvjyothsna]

Github user dvjyothsna commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r180238126
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -289,14 +314,94 @@
   });
   }
   }
-  
+
+  function remoteShutdown(button,host) {
+  var url = location.protocol + "//" + host + 
"/gracefulShutdown";
+  var result = $.ajax({
+type: 'POST',
+url: url,
+contentType : 'text/plain',
+complete: function(data) {
+alert(data.responseJSON["response"]);
+button.prop('disabled',true).css('opacity',0.5);
+}
+  });
+  }
+  
   function getRequestUrl(requestPath) {
 var protocol = location.protocol;
 var host = location.host;
 var url = protocol + "//" + host + requestPath;
 return url;
   }
-
+
+  //Iterates through all the nodes for update
+  function reloadMetrics() {
+  for (i = 1; i <= size; i++) {
+  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+  updateMetricsHtml(address, httpPort, i);
+  }
+  }
+
+  //Update memory
+  //TODO: HTTPS?
+  function updateMetricsHtml(drillbit,webport,idx) {
+var result = $.ajax({
+  type: 'GET',
--- End diff --

I think its safe to not make any remote ajax requests until we figure out a 
way to perform them with BOTH ssl and auth enabled. The user might not notice 
the ajax failures and assume that we are displaying wrong info (The default 
values i.e., zero).


---

[GitHub] drill pull request #1202: DRILL-6311: No logging information in drillbit.log...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1202#discussion_r180226298
  
--- Diff: exec/vector/pom.xml ---
@@ -69,6 +69,7 @@
   org.apache.drill
   drill-common
   ${project.version}
+  test
--- End diff --

How it worked before? In the commit that introduced the test dependency, 
there is no test categorization.


---

[GitHub] drill pull request #1203: DRILL-6289: Cluster view should show more relevant...

[kkhatua]

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r180226226
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -289,14 +314,94 @@
   });
   }
   }
-  
+
+  function remoteShutdown(button,host) {
+  var url = location.protocol + "//" + host + 
"/gracefulShutdown";
+  var result = $.ajax({
+type: 'POST',
+url: url,
+contentType : 'text/plain',
+complete: function(data) {
+alert(data.responseJSON["response"]);
+button.prop('disabled',true).css('opacity',0.5);
+}
+  });
+  }
+  
   function getRequestUrl(requestPath) {
 var protocol = location.protocol;
 var host = location.host;
 var url = protocol + "//" + host + requestPath;
 return url;
   }
-
+
+  //Iterates through all the nodes for update
+  function reloadMetrics() {
+  for (i = 1; i <= size; i++) {
+  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+  updateMetricsHtml(address, httpPort, i);
+  }
+  }
+
+  //Update memory
+  //TODO: HTTPS?
+  function updateMetricsHtml(drillbit,webport,idx) {
+var result = $.ajax({
+  type: 'GET',
--- End diff --

Good catch. I believe we'll get invalid certificate errors since the 
certificate exceptions most likely are not added. 
So the question I have then is whether we should disable making **any** 
remote HTTPS calls at all, or take a chance with the hope that the invalid 
certificate +might+ have been added? 


---

[GitHub] drill pull request #1202: DRILL-6311: No logging information in drillbit.log...

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1202#discussion_r180205787
  
--- Diff: exec/vector/pom.xml ---
@@ -69,6 +69,7 @@
   org.apache.drill
   drill-common
   ${project.version}
+  test
--- End diff --

@vrozov JUnit supports including or excluding tests based on categories. 
Categories are just marker interfaces that are passed to the `@Category` 
annotation. For example a test class annotated with `@Category(SlowTest.Class)` 
will be excluded from a Travis run since Travis excludes the SlowTest category. 
All the test categories needed to be included in a common artifact since all 
the drill submodules require the test categories. So the test categories were 
added to drill-common. 

If drill-common is not included mvn would fail with an error when running 
tests marked with `@Category(SlowTest.class)` since JUnit would not be able to 
find the SlowTest class.


---

[GitHub] drill pull request #1203: DRILL-6289: Cluster view should show more relevant...

[dvjyothsna]

Github user dvjyothsna commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r180196490
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -289,14 +314,94 @@
   });
   }
   }
-  
+
+  function remoteShutdown(button,host) {
+  var url = location.protocol + "//" + host + 
"/gracefulShutdown";
+  var result = $.ajax({
+type: 'POST',
+url: url,
+contentType : 'text/plain',
+complete: function(data) {
+alert(data.responseJSON["response"]);
+button.prop('disabled',true).css('opacity',0.5);
+}
+  });
+  }
+  
   function getRequestUrl(requestPath) {
 var protocol = location.protocol;
 var host = location.host;
 var url = protocol + "//" + host + requestPath;
 return url;
   }
-
+
+  //Iterates through all the nodes for update
+  function reloadMetrics() {
+  for (i = 1; i <= size; i++) {
+  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+  updateMetricsHtml(address, httpPort, i);
+  }
+  }
+
+  //Update memory
+  //TODO: HTTPS?
+  function updateMetricsHtml(drillbit,webport,idx) {
+var result = $.ajax({
+  type: 'GET',
--- End diff --

Will the ajax call works in case of HTTPS? or will it throw invalid 
certificate error since the certificate exception is not yet added for that 
host?


---

[jira] [Created] (DRILL-6315) Postgres supports jsonb datatype but drill doesn't

[Ashirvad Gupta (JIRA)]

Ashirvad Gupta created DRILL-6315:
-

 Summary: Postgres supports jsonb datatype but drill doesn't
 Key: DRILL-6315
 URL: https://issues.apache.org/jira/browse/DRILL-6315
 Project: Apache Drill
  Issue Type: Wish
  Components: SQL Parser
Affects Versions: 1.13.0
Reporter: Ashirvad Gupta
 Fix For: Future, 2.0.0


Postgres supports jsonb datatype using *postgresql-42.0.0.jar* but drill 
doesn't support jsonb data type. It will be good if jsonb datatype is supported 
in drill.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] drill issue #1203: DRILL-6289: Cluster view should show more relevant inform...

[kkhatua]

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1203
  
@arina-ielchiieva You're correct about the inability to shutdown other 
Drillbits from the WebUI. But, from what @dvjyothsna explained, it seems that 
this was the result of a technical hurdle with HTTP**S** protocol. My 
understanding was that issuing shutdown to secure remote nodes won't work 
because of lack of certificates for those nodes, making it difficult for the 
remote Drillbit to authenticate the shutdown command. Keeping that in mind, the 
SHUTDOWN buttons would appear only for unsecured  Drillbits, i.e. when 
Authentication is **_not_** enabled.

https://github.com/kkhatua/drill/blob/dd507ea95a7d0ef4904092ef4fc43ee3b44ef058/exec/java-exec/src/main/resources/rest/index.ftl#L100
For setups requiring Authentication, the shutdown button would not appear.



---