[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93741438
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/client/DrillClientSystemTest.java
 ---
@@ -17,6 +17,8 @@
  */
 package org.apache.drill.exec.client;
 
+import static org.junit.Assert.assertFalse;
+
--- End diff --

Remove unnecessary newline introduced


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93741386
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/util/Utilities.java ---
@@ -47,7 +47,7 @@ public static String 
getFileNameForQueryFragment(FragmentContext context, String
* QueryContextInformation is derived from the current state of the 
process.
*
* @param defaultSchemaName
-   * @return
+   * @return QueryContextInformation
--- End diff --

Revert the change as this has got nothing to do with this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93740038
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -319,4 +319,3 @@ message Jar {
   optional string name = 1;
   repeated string function_signature = 2;
 }
-
--- End diff --

Remove newline introduced


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93741547
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/UserProtos.java ---
@@ -25001,7 +25001,7 @@ public Builder 
setUpdatability(org.apache.drill.exec.proto.UserProtos.ColumnUpda
*
* 
*
-   * Defaults to READ_ONLU
+   * Defaults to READ_ONLY
--- End diff --

Revert changes as this has got nothing to do with this PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5158) BootStrapContext has scan thread size set to large for embedded

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5158:
--

 Summary: BootStrapContext has scan thread size set to large for 
embedded
 Key: DRILL-5158
 URL: https://issues.apache.org/jira/browse/DRILL-5158
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.9.0
Reporter: Paul Rogers
Assignee: Parth Chandra
Priority: Trivial


The Parquet async project added a new config setting: the number of scan 
threads. But, the hard-coded minimum is too large for an embedded Drillbit on a 
Mac used for unit tests.

In {{BootStrapContext}}:

{code}
  private static final int MIN_SCAN_THREADPOOL_SIZE = 8; // Magic num
  ...
final int numScanThreads = (int) 
(config.getDouble(ExecConstants.SCAN_THREADPOOL_SIZE));
...
final int scanThreadPoolSize =
MIN_SCAN_THREADPOOL_SIZE > numScanThreads ? MIN_SCAN_THREADPOOL_SIZE : 
numScanThreads;
{code}

Effectively, this code says you can have as small a thread pool as you like as 
long as it is 8 or greater.

Empirically, a thread pool size of 2 (set by altering the above code) caused 
the Drillbit to hang. A size of 4 allowed the Drillbit to run.

In theory, a size of 0 should be fine as long as the code does not use Parquet 
(or use the async reader). A pool size of 2 should work, but be slow (compared 
to a larger size.)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93739906
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -343,7 +343,6 @@ public BitToUserHandshake 
getHandshakeResponse(UserToBitHandshake inbound) throw
   if (inbound.getRpcVersion() != UserRpcConfig.RPC_VERSION) {
 final String errMsg = String.format("Invalid rpc version. 
Expected %d, actual %d.",
 UserRpcConfig.RPC_VERSION, inbound.getRpcVersion());
-
--- End diff --

Remove newline introduced


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93780471
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java ---
@@ -141,7 +141,7 @@ public FragmentContext(final DrillbitContext dbContext, 
final PlanFragment fragm
 this.accountingUserConnection = new 
AccountingUserConnection(connection, sendingAccountor, statusHandler);
 this.fragment = fragment;
 this.funcRegistry = funcRegistry;
-contextInformation = new ContextInformation(fragment.getCredentials(), 
fragment.getContext());
+contextInformation = new ContextInformation(fragment.getCredentials(), 
fragment.getContext(), connection.getSession().getSessionId());
--- End diff --

accessing `connection` results into NPE for non-root fragment contexts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #685: Drill 5043: Function that returns a unique id per s...

[nagarajanchinnasamy]

Github user nagarajanchinnasamy commented on a diff in the pull request:

https://github.com/apache/drill/pull/685#discussion_r93743678
  
--- Diff: protocol/src/main/protobuf/User.proto ---
@@ -361,7 +361,7 @@ message ResultColumnMetadata {
   optional ColumnSearchability searchability = 13;
 
   /*
-   * Defaults to READ_ONLY
+   * Defaults to READ_ONLU
--- End diff --

Revert the change


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #705: DRILL-5159: Make sure Drill's ProjectMergeRule oper...

[jinfengni]

GitHub user jinfengni opened a pull request:

https://github.com/apache/drill/pull/705

DRILL-5159: Make sure Drill's ProjectMergeRule operate on RelNodes wi…

…th same convention trait.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jinfengni/incubator-drill DRILL-5159

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/705.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #705


commit 02af67c6223689e952cbe4775ccc6dec28269ada
Author: Jinfeng Ni 
Date:   2016-12-22T02:00:46Z

DRILL-5159: Make sure Drill's ProjectMergeRule operate on RelNodes with 
same convention trait.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DRILL-5161) Several TestConvertFunctions tests produce scalar replacement errors

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5161:
--

 Summary: Several TestConvertFunctions tests produce scalar 
replacement errors
 Key: DRILL-5161
 URL: https://issues.apache.org/jira/browse/DRILL-5161
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Paul Rogers
Priority: Minor


Run the unit test {{TestConvertFunctions}}. The {{testBinaryString}} test emits 
an ASM error (see below), though the test succeeds. In addition, the test 
{{testConvertFromConvertToInt}} also emits the error if the line to turn off 
scalar replacement is commented out (see test.)

Another bug complains about the stack trace displayed when scalar replacement 
fails.

This bug says that we should either not use scalar replacement if we can 
predict it will fail, or disable scalar replacement altogether so that we get 
clean tests. (Work elsewhere suggested that the Java compiler and runtime 
already to scalar replacement for us in the latest Java versions.)

Stack trace:

{code}
unning org.apache.drill.exec.physical.impl.TestConvertFunctions#testBinaryString
org.objectweb.asm.tree.analysis.AnalyzerException: Error at instruction 175: 
Expected an object reference, but found .
at org.objectweb.asm.tree.analysis.Analyzer.analyze(Analyzer.java:294)
at 
org.objectweb.asm.util.CheckMethodAdapter$1.visitEnd(CheckMethodAdapter.java:450)
at org.objectweb.asm.MethodVisitor.visitEnd(MethodVisitor.java:877)
...
at org.objectweb.asm.tree.ClassNode.accept(ClassNode.java:412)
at 
org.apache.drill.exec.compile.MergeAdapter.getMergedClass(MergeAdapter.java:227)
at 
org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:280)
at 
org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:226)
at 
org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:128)
at 
org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:1)
...
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5159) ProjectMergeRule in Drill should operate on RelNodes with same convention trait.

[Jinfeng Ni (JIRA)]

Jinfeng Ni created DRILL-5159:
-

 Summary: ProjectMergeRule in Drill should operate on RelNodes with 
same convention trait.
 Key: DRILL-5159
 URL: https://issues.apache.org/jira/browse/DRILL-5159
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Jinfeng Ni


Drill extended version of  Calcite's ProjectMergeRule is used in a 
VolcanoPlanner where RelNodes with different convention could match with this 
rule. 

For instance, we could see this rule could be invoked when a DrillProject on 
top of a LogicalProject. Also, since the output RelNode is built from the 
default Project RelFactory, such rule execution could end up with a 
LogicalProject.

{code}
DrillProjecttransform  
\  ===>   LogicalProject
LogicalProject
{code}
 
This leads to un-necessary rule execution, or in certain case could lead to an 
infinite loop.  

The proposed fix is to check matched RelNodes to make sure that they do have 
Calcite Logical convention. That way, both inputs and output of this rule would 
have same convention trait.  This should reduce planning time, and avoid the 
possiblity of loop.







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5160) Memory leak in Parquet async reader when Snappy fails

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-5160:
--

 Summary: Memory leak in Parquet async reader when Snappy fails
 Key: DRILL-5160
 URL: https://issues.apache.org/jira/browse/DRILL-5160
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Paul Rogers
Assignee: Parth Chandra


See the details in DRILL-5157. The Parquet async reader uses the Snappy 
library. If the call into Snappy fails (in my case, due to missing dependency 
management in the Drill pom.xml file), the Parquet reader fails (as it should.)

Unfortunately, the Parquet reader leaks memory which cause 
{{IllegalStateException}} errors in production, assertion failures in unit 
tests.

To reproduce this the easy way (to avoid the need to undo the fix for 
DRILL-5157 in Snappy dependencies), use Drill's exception injector in inject an 
exception at the line where we call into Snappy (See DRILL-5157 for details.)

Set a breakpoint exception for {{IllegalStateException}} you will see the 
memory leak. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)