[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449406#comment-16449406
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user ilooner commented on the issue:
https://github.com/apache/drill/pull/1234
@vrozov addressed comments
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:44)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> [junit-4.11.jar:na]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> [junit-4.11.jar:na]
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> [junit-4.11.jar:na]
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> [junit-rt.jar:na]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449332#comment-16449332
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1234#discussion_r183611338
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/bson/TestBsonRecordReader.java
---
@@ -49,17 +50,20 @@
import org.junit.BeforeClass;
import org.junit.Test;
-public class TestBsonRecordReader extends BaseTestQuery {
+public class TestBsonRecordReader {
+ private static BufferAllocator allocator;
private static VectorContainerWriter writer;
private static TestOutputMutator mutator;
+ private static DrillBuf buffer;
private static BsonRecordReader bsonReader;
@BeforeClass
public static void setUp() {
-BufferAllocator bufferAllocator = getDrillbitContext().getAllocator();
-mutator = new TestOutputMutator(bufferAllocator);
+allocator = new RootAllocator(100_000_000);
--- End diff --
After testing just now, the lowest I could push this was 9,000,000 bytes.
Going to 8,000,000 bytes causes an OOM. It looks like the BsonRecordReader
allocates a bunch of value vectors. I will adjust this to 9,000,000.
```
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
buffer of size 8192 due to memory limit (80). Current allocation: 795648
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:236)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:211)
at
org.apache.drill.exec.vector.UInt1Vector.reallocRaw(UInt1Vector.java:262)
at
org.apache.drill.exec.vector.UInt1Vector.reAlloc(UInt1Vector.java:251)
at
org.apache.drill.exec.vector.UInt1Vector$Mutator.setSafe(UInt1Vector.java:480)
at
org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(NullableVarCharVector.java:608)
at
org.apache.drill.exec.vector.complex.impl.NullableVarCharWriterImpl.write(NullableVarCharWriterImpl.java:110)
at
org.apache.drill.exec.store.bson.BsonRecordReader.writeString(BsonRecordReader.java:276)
at
org.apache.drill.exec.store.bson.BsonRecordReader.writeBinary(BsonRecordReader.java:205)
at
org.apache.drill.exec.store.bson.BsonRecordReader.writeToListOrMap(BsonRecordReader.java:117)
at
org.apache.drill.exec.store.bson.BsonRecordReader.write(BsonRecordReader.java:75)
at
org.apache.drill.exec.store.bson.TestBsonRecordReader.testBinaryTypes(TestBsonRecordReader.java:244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
```
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(A
[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449308#comment-16449308
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1234#discussion_r183608559
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/bson/TestBsonRecordReader.java
---
@@ -272,6 +276,9 @@ public static void cleanUp() {
} catch (Exception e) {
}
+
+buffer.close();
--- End diff --
Currently close() just calls release(). I will update to use release(), but
in general what is the best practice regarding when to call close() and when to
call release()?
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:44)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> [junit-4.11.jar:na]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> [junit-4.11.jar:na]
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> [junit-4.11.jar:na]
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWi
[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449307#comment-16449307
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/drill/pull/1234#discussion_r183608064
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/bson/TestBsonRecordReader.java
---
@@ -49,17 +50,20 @@
import org.junit.BeforeClass;
import org.junit.Test;
-public class TestBsonRecordReader extends BaseTestQuery {
+public class TestBsonRecordReader {
+ private static BufferAllocator allocator;
--- End diff --
JUnit requires tests annotated with @BeforeClass and @AfterClass to be
static so these variables have to be static. Currently the Drill unit tests do
not support concurrent execution of methods within a test class within the same
process. Drill has surefire launch multiple test processes and the the test
classes to execute are divided among the test processes. Within each forked
surefire process tests are executed sequentially, so this will not be an issue.
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:44)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> [ju
[jira] [Commented] (DRILL-143) Support CGROUPs resource management
[
https://issues.apache.org/jira/browse/DRILL-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449301#comment-16449301
]
ASF GitHub Bot commented on DRILL-143:
--
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1239#discussion_r183607110
--- Diff: distribution/src/resources/yarn-drillbit.sh ---
@@ -110,6 +114,36 @@
# Enables Java GC logging. Passed from the drill.yarn.drillbit.log-gc
# garbage collection option.
+### Function to enforce CGroup (Refer local drillbit.sh)
+check_and_enforce_cgroup(){
+dbitPid=$1;
+kill -0 $dbitPid
+if [ $? -gt 0 ]; then
+ echo "ERROR: Failed to add Drillbit to CGroup ( $DRILLBIT_CGROUP )
for 'cpu'. Ensure that the Drillbit ( pid=$dbitPid ) started up." >&2
+ exit 1
+fi
+SYS_CGROUP_DIR=${SYS_CGROUP_DIR:-"/sys/fs/cgroup"}
+if [ -f $SYS_CGROUP_DIR/cpu/$DRILLBIT_CGROUP/cgroup.procs ]; then
+ echo $dbitPid > $SYS_CGROUP_DIR/cpu/$DRILLBIT_CGROUP/cgroup.procs
+ # Verify Enforcement
+ cgroupStatus=`grep -w $pid
$SYS_CGROUP_DIR/cpu/${DRILLBIT_CGROUP}/cgroup.procs`
+ if [ -z "$cgroupStatus" ]; then
--- End diff --
I'm confused: Is this checking for $dbitPid (in cgroup.procs) or for $pid ?
In case the former, then need to negate the following "-z" condition.
> Support CGROUPs resource management
> ---
>
> Key: DRILL-143
> URL: https://issues.apache.org/jira/browse/DRILL-143
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
> Attachments: 253ce178-ddeb-e482-cd64-44ab7284ad1c.sys.drill
>
>
> For the purpose of playing nice on clusters that don't have YARN, we should
> write up configuration and scripts to allows users to run Drill next to
> existing workloads without sharing resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6335) Refactor row set abstractions to prepare for unions
[ https://issues.apache.org/jira/browse/DRILL-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-6335: --- Labels: ready-to-commit (was: ) Reviewer: Padma Penumarthy > Refactor row set abstractions to prepare for unions > --- > > Key: DRILL-6335 > URL: https://issues.apache.org/jira/browse/DRILL-6335 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > The row set abstractions will eventually support unions and lists. The > changes to support these types are extensive. This PR introduces refactoring > that puts the pieces in the correct location to allow for inserting those > additional types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[
https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449124#comment-16449124
]
ASF GitHub Bot commented on DRILL-6281:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1238#discussion_r183578626
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FooterGatherer.java
---
@@ -66,8 +69,8 @@ private static void checkMagicBytes(FileStatus status,
byte[] data, int offset)
}
public static List getFooters(final Configuration conf,
List statuses, int parallelism) throws IOException {
-final List> readers = Lists.newArrayList();
-List foundFooters = Lists.newArrayList();
+final List> readers = new ArrayList<>();
+final List foundFooters = new ArrayList<>();
--- End diff --
Please see Guava Java doc
[Lists.newArrayList()](http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/collect/Lists.html#newArrayList--):
> **Note for Java 7 and later:** this method is now unnecessary and should
be treated as deprecated. Instead, use the ArrayList
[constructor](https://docs.oracle.com/javase/9/docs/api/java/util/ArrayList.html?is-external=true#ArrayList--)
directly, taking advantage of the new ["diamond" syntax](http://goo.gl/iz2Wi).
> Refactor TimedRunnable
> --
>
> Key: DRILL-6281
> URL: https://issues.apache.org/jira/browse/DRILL-6281
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[
https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449111#comment-16449111
]
ASF GitHub Bot commented on DRILL-6281:
---
Github user HanumathRao commented on a diff in the pull request:
https://github.com/apache/drill/pull/1238#discussion_r183576419
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FooterGatherer.java
---
@@ -66,8 +69,8 @@ private static void checkMagicBytes(FileStatus status,
byte[] data, int offset)
}
public static List getFooters(final Configuration conf,
List statuses, int parallelism) throws IOException {
-final List> readers = Lists.newArrayList();
-List foundFooters = Lists.newArrayList();
+final List> readers = new ArrayList<>();
+final List foundFooters = new ArrayList<>();
--- End diff --
Any specific reason to not use Lists.newArrayList?
> Refactor TimedRunnable
> --
>
> Key: DRILL-6281
> URL: https://issues.apache.org/jira/browse/DRILL-6281
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[
https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449108#comment-16449108
]
ASF GitHub Bot commented on DRILL-6281:
---
Github user HanumathRao commented on a diff in the pull request:
https://github.com/apache/drill/pull/1238#discussion_r183576093
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.drill.common.exceptions.UserException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * Class used to allow parallel executions of tasks in a simplified way.
Also maintains and reports timings of task completion.
+ * TODO: look at switching to fork join.
+ * @param The time value that will be returned when the task is
executed.
+ */
+public abstract class TimedCallable implements Callable {
+ private static final Logger logger =
LoggerFactory.getLogger(TimedCallable.class);
+
+ private static long TIMEOUT_PER_RUNNABLE_IN_MSECS = 15000;
+
+ private volatile long startTime = 0;
+ private volatile long executionTime = -1;
+
+ private static class FutureMapper implements Function, V> {
+int count;
+Throwable throwable = null;
+
+private void setThrowable(Throwable t) {
+ if (throwable == null) {
+throwable = t;
+ } else {
+throwable.addSuppressed(t);
+ }
+}
+
+@Override
+public V apply(Future future) {
+ Preconditions.checkState(future.isDone());
+ if (!future.isCancelled()) {
+try {
+ count++;
+ return future.get();
+} catch (InterruptedException e) {
+ // there is no wait as we are getting result from the
completed/done future
+ logger.error("Unexpected exception", e);
+ throw UserException.internalError(e)
+ .message("Unexpected exception")
+ .build(logger);
+} catch (ExecutionException e) {
+ setThrowable(e.getCause());
+}
+ } else {
+setThrowable(new CancellationException());
+ }
+ return null;
+}
+ }
+
+ private static class Statistics implements Consumer>
{
+final long start = System.nanoTime();
+final Stopwatch watch = Stopwatch.createStarted();
+long totalExecution = 0;
+long maxExecution = 0;
+int startedCount = 0;
+private int doneCount = 0;
+// measure thread creation times
+long earliestStart = Long.MAX_VALUE;
+long latestStart = 0;
+long totalStart = 0;
+
+@Override
+public void accept(TimedCallable task) {
+ long threadStart = task.getStartTime(TimeUnit.NANOSECONDS) - start;
+ if (threadStart >= 0) {
+startedCount++;
+earliestStart = Math.min(earliestStart, threadStart);
+latestStart = Math.max(latestStart, threadStart);
+t
[jira] [Commented] (DRILL-6335) Refactor row set abstractions to prepare for unions
[ https://issues.apache.org/jira/browse/DRILL-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449092#comment-16449092 ] ASF GitHub Bot commented on DRILL-6335: --- Github user ppadma commented on the issue: https://github.com/apache/drill/pull/1218 LGTM. +1 > Refactor row set abstractions to prepare for unions > --- > > Key: DRILL-6335 > URL: https://issues.apache.org/jira/browse/DRILL-6335 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.14.0 > > > The row set abstractions will eventually support unions and lists. The > changes to support these types are extensive. This PR introduces refactoring > that puts the pieces in the correct location to allow for inserting those > additional types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6202) Deprecate usage of IndexOutOfBoundsException to re-alloc vectors
[ https://issues.apache.org/jira/browse/DRILL-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449061#comment-16449061 ] ASF GitHub Bot commented on DRILL-6202: --- Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1144#discussion_r183571786 --- Diff: src/main/resources/checkstyle-config.xml --- @@ -30,10 +30,15 @@ + --- End diff -- IMO the same applies to all run-time exceptions (NPE, CCE, IOBE) and likely conversion for NPE is the same as for IOBE. Note that most likely the root cause of an IOBE or NPE is a bug and not an end user error, misconfiguration, invalid input or a lack of resources, so end user won't be able to "fix" NPE or IOBE. > Deprecate usage of IndexOutOfBoundsException to re-alloc vectors > > > Key: DRILL-6202 > URL: https://issues.apache.org/jira/browse/DRILL-6202 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > As bounds checking may be enabled or disabled, using > IndexOutOfBoundsException to resize vectors is unreliable. It works only when > bounds checking is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449021#comment-16449021
]
Kunal Khatua commented on DRILL-5977:
-
[~aravi5] , when do you think this would be ready for a review?
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Abhishek Ravi
>Priority: Major
> Fix For: 1.14.0
>
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449018#comment-16449018
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1234#discussion_r183563218
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/bson/TestBsonRecordReader.java
---
@@ -49,17 +50,20 @@
import org.junit.BeforeClass;
import org.junit.Test;
-public class TestBsonRecordReader extends BaseTestQuery {
+public class TestBsonRecordReader {
+ private static BufferAllocator allocator;
private static VectorContainerWriter writer;
private static TestOutputMutator mutator;
+ private static DrillBuf buffer;
private static BsonRecordReader bsonReader;
@BeforeClass
public static void setUp() {
-BufferAllocator bufferAllocator = getDrillbitContext().getAllocator();
-mutator = new TestOutputMutator(bufferAllocator);
+allocator = new RootAllocator(100_000_000);
--- End diff --
Is it necessary to reserve 100MB for allocator? Does it allocate more than
1K?
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:44)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runner
[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449016#comment-16449016
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1234#discussion_r183562766
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/bson/TestBsonRecordReader.java
---
@@ -272,6 +276,9 @@ public static void cleanUp() {
} catch (Exception e) {
}
+
+buffer.close();
--- End diff --
buffer.release()?
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:44)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> [junit-4.11.jar:na]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> [junit-4.11.jar:na]
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> [junit-4.11.jar:na]
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> [junit-rt.jar:na]
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
[jira] [Commented] (DRILL-5927) Root allocator consistently Leaks a buffer in unit tests
[
https://issues.apache.org/jira/browse/DRILL-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449017#comment-16449017
]
ASF GitHub Bot commented on DRILL-5927:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1234#discussion_r183564004
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/bson/TestBsonRecordReader.java
---
@@ -49,17 +50,20 @@
import org.junit.BeforeClass;
import org.junit.Test;
-public class TestBsonRecordReader extends BaseTestQuery {
+public class TestBsonRecordReader {
+ private static BufferAllocator allocator;
--- End diff --
What is a reason to define `BufferAllocator` and other fields as static?
What if tests are executed in parallel?
> Root allocator consistently Leaks a buffer in unit tests
>
>
> Key: DRILL-5927
> URL: https://issues.apache.org/jira/browse/DRILL-5927
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Minor
> Fix For: 1.14.0
>
>
> TestBsonRecordReader consistently poduces this exception when running on my
> laptop
> {code}
> 13:09:15.777 [main] ERROR o.a.d.exec.server.BootStrapContext - Error while
> closing
> java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding
> buffers allocated (1).
> Allocator(ROOT) 0/1024/10113536/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[79] allocator: ROOT), isOwning: true, size: 1024, references: 1,
> life: 340912804170064..0, allocatorManager: [71, life: 340912803759189..0]
> holds 1 buffers.
> DrillBuf[106], udle: [72 0..1024]
> reservations: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at
> org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:256)
> ~[classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> [classes/:na]
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> [classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:205)
> [classes/:na]
> at org.apache.drill.BaseTestQuery.closeClient(BaseTestQuery.java:315)
> [test-classes/:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_144]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_144]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.11.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> [junit-4.11.jar:na]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:44)
> [jmockit-1.3.jar:na]
> at
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
> [jmockit-1.3.jar:na]
> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_144]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
> at
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
> [jmockit-1.3.jar:na]
> at
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
> [jmockit-1.3.jar:na]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> [junit-4.11.jar:na]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> [junit-4.11.jar:na]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> [junit-4.11.jar:na]
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> [junit-4.11.jar:na]
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> [junit-rt.jar:na]
> at
> com.intellij.rt.exec
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448995#comment-16448995
]
ASF GitHub Bot commented on DRILL-6336:
---
Github user vrozov commented on the issue:
https://github.com/apache/drill/pull/1235
There is no need to expose implementation details as part of the class API.
Whether `DebugStringBuilder` uses `PrintWriter.print()` or something else to
implement `append()` must be hidden from `DebugStringBuilder` consumers. Method
name should never depend on details of implementation as the implementation may
change, but API should not.
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6348: - Reviewer: Parth Chandra > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6249: - Fix Version/s: 1.14.0 > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6347) Inconsistent method name "field".
[
https://issues.apache.org/jira/browse/DRILL-6347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448863#comment-16448863
]
ASF GitHub Bot commented on DRILL-6347:
---
Github user vrozov commented on the issue:
https://github.com/apache/drill/pull/1236
IMO, neither `append` or `appendField` is a good choice (otherwise it is
necessary to change `startNode/endNode` to `appendStart/EndNode`). It is either
`field` or `visitField` and should follow visitor design pattern. It may be
also good to change `listField` to the same name used for `Boolean` and
`String`.
> Inconsistent method name "field".
> -
>
> Key: DRILL-6347
> URL: https://issues.apache.org/jira/browse/DRILL-6347
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Assignee: KuiLIU
>Priority: Major
> Fix For: 1.14.0
>
>
> The following method is names as "field", but the method is mainly doing
> appending. So that, rename the method as "append" should be better.
> {code:java}
> private void field(String label, String value) {
> indent();
> out.append(label)
> .append(" = ")
> .append(value)
> .append("\n");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6202) Deprecate usage of IndexOutOfBoundsException to re-alloc vectors
[ https://issues.apache.org/jira/browse/DRILL-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448836#comment-16448836 ] ASF GitHub Bot commented on DRILL-6202: --- Github user parthchandra commented on a diff in the pull request: https://github.com/apache/drill/pull/1144#discussion_r183530901 --- Diff: src/main/resources/checkstyle-config.xml --- @@ -30,10 +30,15 @@ + --- End diff -- I think IOBE should be caught and converted to a UserException so that the end user does not get an incomprehensible error message. > Deprecate usage of IndexOutOfBoundsException to re-alloc vectors > > > Key: DRILL-6202 > URL: https://issues.apache.org/jira/browse/DRILL-6202 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > As bounds checking may be enabled or disabled, using > IndexOutOfBoundsException to resize vectors is unreliable. It works only when > bounds checking is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
[
https://issues.apache.org/jira/browse/DRILL-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448832#comment-16448832
]
Marc Prud'hommeaux commented on DRILL-6349:
---
This patch may be sufficient to get the driver to load on Java 9+:
https://github.com/apache/drill/compare/master...mprudhom:patch-1
> Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
> ---
>
> Key: DRILL-6349
> URL: https://issues.apache.org/jira/browse/DRILL-6349
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
> Environment: 16:23 apache-drill-1.13.0$ uname -a
> Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
> 2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
>
>Reporter: Marc Prud'hommeaux
>Priority: Major
>
> I'm surprised to be unable to find this issue logged elsewhere, but a quick
> search yields nothing. Trying to connect with the JDBC driver raises a
> NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
> understandable that the full drillbit (or its many dependencies) may have
> difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
> driver should be able to be loaded by any JVM.
>
> Looking at
> ./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
> appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
> called lazily from the one place that uses it: getMaxDirectMemory()
>
>
>
> {{}}{{16:21 apache-drill-1.13.0$
> JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk
> ./bin/drill-localhost }}
> ***Connecting to jdbc:drill:drillbit=localhost*
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> (file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
> to field java.nio.Buffer.address
> WARNING: Please consider reporting this to the maintainers of
> org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {{java.lang.NoClassDefFoundError: sun/misc/VM}}
> {{ at
> org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
> {{ at
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
> {{ at
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
> {{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
> {{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
> {{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
> {{ at sqlline.Commands.connect(Commands.java:1083)}}
> {{ at sqlline.Commands.connect(Commands.java:1015)}}
> {{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)}}
> {{ at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
> {{ at
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
> {{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
> {{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
> {{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
> {{ at sqlline.SqlLine.start(SqlLine.java:375)}}
> {{ at sqlline.SqlLine.main(SqlLine.java:268)}}
> {{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
> {{ at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
> {{ at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
> {{ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)}}
> {{... 20 more}}
> apache drill 1.13.0
> "just drill it"
> 0: jdbc:drill:drillbit=localhost>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6173) Support transitive closure during filter push down and partition pruning
[
https://issues.apache.org/jira/browse/DRILL-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448828#comment-16448828
]
ASF GitHub Bot commented on DRILL-6173:
---
Github user amansinha100 commented on the issue:
https://github.com/apache/drill/pull/1216
+1
> Support transitive closure during filter push down and partition pruning
>
>
> Key: DRILL-6173
> URL: https://issues.apache.org/jira/browse/DRILL-6173
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.12.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> There is Calcite rule JoinPushTransitivePredicatesRule but it does not work
> in Drill.
> Applying it in Drill will allow for equi-join queries to push filter
> condition from one table to another:
> {code:sql}
> select *
> from A, B
> where
> A.id = B.id and
> B.id = 100
> {code}
> In that case it is possible that Scan operator for A table will not scan all
> data.
> For table A it can lead for applying:
> 1. [Partition pruning for Hive or file system Parquet
> tables|https://drill.apache.org/docs/partition-pruning-introduction/]
> 2. [Parquet filter
> pushdown|https://drill.apache.org/docs/parquet-filter-pushdown/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6349) Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
Marc Prud'hommeaux created DRILL-6349:
-
Summary: Drill JDBC driver fails on Java 1.9+ with
NoClassDefFoundError: sun/misc/VM
Key: DRILL-6349
URL: https://issues.apache.org/jira/browse/DRILL-6349
Project: Apache Drill
Issue Type: Bug
Components: Client - JDBC
Affects Versions: 1.13.0
Environment: 16:23 apache-drill-1.13.0$ uname -a
Darwin bix.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar 5 22:24:32 PST
2018; root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
Reporter: Marc Prud'hommeaux
I'm surprised to be unable to find this issue logged elsewhere, but a quick
search yields nothing. Trying to connect with the JDBC driver raises a
NoClassDefFoundError for sun.misc.VM, which was removed in Java 9. It is
understandable that the full drillbit (or its many dependencies) may have
difficult-to-extract dependencies on some sun.misc.* classes, but the JDBC
driver should be able to be loaded by any JVM.
Looking at
./common/src/main/java/org/apache/drill/common/config/DrillConfig.java, it
appears that a quick workaround could be to make sun.misc.VM.maxDirectMemory
called lazily from the one place that uses it: getMaxDirectMemory()
{{}}{{16:21 apache-drill-1.13.0$
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk ./bin/drill-localhost
}}
***Connecting to jdbc:drill:drillbit=localhost*
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
(file:/Users/marc/Downloads/apache-drill-1.13.0/jars/3rdparty/avatica-1.10.0.jar)
to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of
org.apache.calcite.avatica.com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
{{java.lang.NoClassDefFoundError: sun/misc/VM}}
{{ at
org.apache.drill.common.config.DrillConfig.(DrillConfig.java:49)}}
{{ at
org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:155)}}
{{ at
org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)}}
{{ at
org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)}}
{{ at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)}}
{{ at org.apache.drill.jdbc.Driver.connect(Driver.java:72)}}
{{ at sqlline.DatabaseConnection.connect(DatabaseConnection.java:168)}}
{{ at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:214)}}
{{ at sqlline.Commands.connect(Commands.java:1083)}}
{{ at sqlline.Commands.connect(Commands.java:1015)}}
{{ at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)}}
{{ at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
{{ at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
{{ at java.base/java.lang.reflect.Method.invoke(Method.java:564)}}
{{ at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)}}
{{ at sqlline.SqlLine.dispatch(SqlLine.java:742)}}
{{ at sqlline.SqlLine.initArgs(SqlLine.java:528)}}
{{ at sqlline.SqlLine.begin(SqlLine.java:596)}}
{{ at sqlline.SqlLine.start(SqlLine.java:375)}}
{{ at sqlline.SqlLine.main(SqlLine.java:268)}}
{{Caused by: java.lang.ClassNotFoundException: sun.misc.VM}}
{{ at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)}}
{{ at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)}}
{{ at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)}}
{{... 20 more}}
apache drill 1.13.0
"just drill it"
0: jdbc:drill:drillbit=localhost>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6286) Regression: incorrect reference to shutdown in drillbit.log
[
https://issues.apache.org/jira/browse/DRILL-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6286:
-
Labels: ready-to-commit (was: )
> Regression: incorrect reference to shutdown in drillbit.log
> ---
>
> Key: DRILL-6286
> URL: https://issues.apache.org/jira/browse/DRILL-6286
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> drillbit.log refers to shutdown even in cases when no shutdown sequence was
> initiated:
> {noformat}
> 2018-03-16 11:55:52,693 [drill-executor-19] INFO
> o.apache.drill.exec.work.WorkManager - Waiting for 0 queries to complete
> before shutting down
> 2018-03-16 11:55:52,693 [drill-executor-19] INFO
> o.apache.drill.exec.work.WorkManager - Waiting for 3 running fragments to
> complete before shutting down
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3202) Count(*) fails on JSON wrapped up in single array - JSON parsing error
[
https://issues.apache.org/jira/browse/DRILL-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448752#comment-16448752
]
Scott Wilburn commented on DRILL-3202:
--
Is there any workaround for this issue? It's hard to believe that I can't count
objects in a json file using Drill.
> Count(*) fails on JSON wrapped up in single array - JSON parsing error
> --
>
> Key: DRILL-3202
> URL: https://issues.apache.org/jira/browse/DRILL-3202
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
>Affects Versions: 1.0.0
>Reporter: Neeraja
>Assignee: Steven Phillips
>Priority: Major
> Fix For: Future
>
> Attachments: DRILL-3202.patch
>
>
> I have a JSON document as follows.
> [
> {
> "Category": "1,2",
> "Comments": "Total sites: 20, RV sites: 20, Elec sites: 20, Water at
> site, RV Dump, Showers, Flush Toilets, RV Fee: $14, Tent Fee: $14, Elev:
> 545', Tel: 256-577-9619, Nearest town: Muscle Shoals",
> "Latitude": "34.800446",
> "Longitude": "-87.498242",
> "Name": "Alloys Co Park",
> "State": "AL",
> "Type": "cp",
> "URL":
> "http://www.campingroadtrip.com/campgrounds/campground/campground/23478/alabama/colbert-county-alloys-park-campground";
> }
> ]
> Drill has ability to unwrap the array (without user specifying it) and
> perform some SQL operations on it. However count(*) specifically fails on
> these documents.
> 0: jdbc:drill:zk=local> select * from
> dfs.`default`.`/Users/nrentachintala/Downloads/yelp/uspointsofinterestshort.json`
> limit 10;
> +---++---++---++---+--+
> | Category |
>Comments
> | Latitude | Longitude | Name |
> State | Type | URL |
> +---++---++---++---+--+
> | 1,2 | Total sites: 20, RV sites: 20, Elec sites: 20, Water at site, RV
> Dump, Showers, Flush Toilets, RV Fee: $14, Tent Fee: $14, Elev: 545', Tel:
> 256-577-9619, Nearest town: Muscle Shoals | 34.800446 | -87.498242 | Alloys
> Co Park | AL | cp |
> http://www.campingroadtrip.com/campgrounds/campground/campground/23478/alabama/colbert-county-alloys-park-campground
> |
> +---++---++---++---+--+
> 1 row selected (0.197 seconds)
> 0: jdbc:drill:zk=local> select distinct type from
> dfs.`default`.`/Users/nrentachintala/Downloads/yelp/uspointsofinterestshort.json`
> limit 10;
> +---+
> | type |
> +---+
> | cp|
> +---+
> 1 row selected (0.193 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> select count(*) from
> dfs.`default`.`/Users/nrentachintala/Downloads/yelp/uspointsofinterestshort.json`
> limit 10;
> Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a
> record. Current token was START_ARRAY
> File /Users/nrentachintala/Downloads/yelp/uspointsofinterestshort.json
> Record 1
> Fragment 0:0
> [Error Id: 4742f738-1d43-4fef-af48-110065c9dd83 on 172.16.1.82:31010]
> (state=,code=0)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6173) Support transitive closure during filter push down and partition pruning
[
https://issues.apache.org/jira/browse/DRILL-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448738#comment-16448738
]
ASF GitHub Bot commented on DRILL-6173:
---
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/1216#discussion_r183329162
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/planner/logical/TestTransitiveClosure.java
---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.categories.PlannerTest;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.nio.file.Paths;
+
+import static org.junit.Assert.assertEquals;
+
+@Category(PlannerTest.class)
+public class TestTransitiveClosure extends PlanTestBase {
+
+ @BeforeClass
+ public static void setupTestFiles() {
+dirTestWatcher.copyResourceToRoot(Paths.get("join"));
+ }
+
+
+ @Test // CALCITE-2200: (query with infinite loop)
+ public void simpleInfiniteLoop() throws Exception {
+String query = "SELECT t1.department_id FROM cp.`employee.json` t1 " +
+" WHERE t1.department_id IN (SELECT department_id FROM
cp.`department.json` t2 " +
+"WHERE t1.department_id =
t2.department_id " +
+"OR (t1.department_id IS NULL and
t2.department_id IS NULL))";
+int actualRowCount = testSql(query);
+int expectedRowCount = 1155;
+assertEquals("Expected and actual row count should match",
expectedRowCount, actualRowCount);
+
+// TODO: After resolving CALCITE-2257 there will not be Filter with
OR(IS NOT NULL($0), IS NULL($0)) condition
+// Then remove testPlanMatchingPatterns() from this test
+final String[] expectedPlan =
+new String[] {"Filter\\(condition=\\[OR\\(IS NOT NULL\\(\\$0\\),
IS NULL\\(\\$0\\)\\)\\]\\)"};
+final String[] excludedPlan ={};
+testPlanMatchingPatterns(query, expectedPlan, excludedPlan);
+ }
+
+ @Test // CALCITE-2205 (query with infinite loop)
+ public void infiniteLoopWhilePlaningComplexQuery() throws Exception {
--- End diff --
This test and other one from TestTransitiveClosure.java are from functional
test suite.
So we can remove them from unit tests scope.
> Support transitive closure during filter push down and partition pruning
>
>
> Key: DRILL-6173
> URL: https://issues.apache.org/jira/browse/DRILL-6173
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 1.12.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> There is Calcite rule JoinPushTransitivePredicatesRule but it does not work
> in Drill.
> Applying it in Drill will allow for equi-join queries to push filter
> condition from one table to another:
> {code:sql}
> select *
> from A, B
> where
> A.id = B.id and
> B.id = 100
> {code}
> In that case it is possible that Scan operator for A table will not scan all
> data.
> For table A it can lead for applying:
> 1. [Partition pruning for Hive or file system Parquet
> tables|https://drill.apache.org/docs/partition-pruning-introduction/]
> 2. [Parquet filter
> pushdown|https://drill.apache.org/docs/parquet-filter-pushdown/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6346) Create an Official Drill Docker Container
[
https://issues.apache.org/jira/browse/DRILL-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448703#comment-16448703
]
Abhishek Girish commented on DRILL-6346:
I have a working copy of this ready.
*Embedded mode:*
(1) Clone repository
{code:java}
git clone g...@github.com:Agirish/drill-docker.git
{code}
(2) Launch Drill container
{code:java}
cd drill-docker/scripts
./launchDockerContainer.sh drill centos 1.13.0
{code}
(3) Launch Sqlline
{code:java}
/opt/drill/bin/drill-embedded
{code}
*Distributed mode:*
(1) Clone repository
{code:java}
git clone g...@github.com:Agirish/drill-docker.git
{code}
(2) Launch Zookeeper container
{code:java}
cd drill-docker/scripts
./launchDockerContainer.sh zookeeper centos 3.4.10
{code}
Make a note of the ZK container IP address
(3) Launch Drill container
{code:java}
cd drill-docker/scripts
./launchDockerContainer.sh drill centos 1.13.0
{code}
(4) Configure Drillbits
Edit /opt/drill/conf/drill-override.conf. Update the zk.connect string
with the ZK container IP address
(5) Start Drillbits
{code:java}
/opt/drill/bin/drillbit.sh start
{code}
(6) Launch Sqlline
{code:java}
/opt/drill/bin/sqlline -u jdbc:drill:zk=ZK_IP_ADDR:2181
{code}
Repeat steps (3)-(5) to add more Drillbits. I've tested with 2 drillbits and
they successfully join to form a cluster.
> Create an Official Drill Docker Container
> -
>
> Key: DRILL-6346
> URL: https://issues.apache.org/jira/browse/DRILL-6346
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Abhishek Girish
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448593#comment-16448593
]
ASF GitHub Bot commented on DRILL-6242:
---
Github user jiang-wu commented on the issue:
https://github.com/apache/drill/pull/1184
I was out of town last week. Will work on the type change to Java 8
Local[Data|Time|Timestamp] this week and then notify you when it is done.
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Assignee: Jiang Wu
>Priority: Major
> Fix For: 1.14.0
>
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfSecond":940,"secondOfMinute":54,"secondOfDay":78774,"minuteOfHour":52,"minuteOfDay":1312,"hourOfDay":21,"zone":{"fixed":true,"id":"UTC"},"millis":1520977974940,"chronology":{"zone":{"fixed":true,"id":"UTC"}},"afterNow":false,"beforeNow":true,"equalNow":false},"user":"jack"}
> |
> {code}
> We can see that from the above output, when the date field is retrieved as a
> top level column, Drill outputs a logical date value. But when the same
> field is within an object hierarchy, Drill outputs the internal object used
> to hold the date value.
> The expected output is the same display for whether the date field is shown
> as a top level column or when it is within an object hierarchy:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 | {"date":"2018-03-13","user":"jack"} |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5892) Distinguish between states for query statistics exposed via JMX
[
https://issues.apache.org/jira/browse/DRILL-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448552#comment-16448552
]
Kunal Khatua commented on DRILL-5892:
-
Hi [~josiah]
Sorry, I seem to have missed this. Please go ahead and assign this to your
self. As per my knowledge, no one else has started work on this.
Thanks!
> Distinguish between states for query statistics exposed via JMX
> ---
>
> Key: DRILL-5892
> URL: https://issues.apache.org/jira/browse/DRILL-5892
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Priority: Major
> Fix For: Future
>
>
> Currently, the JMX metrics exposed
> {code:java}
> metrics:name=drill.queries.completed
> metrics:name=drill.queries.running
> metrics:name=drill.queries.enqueued
> {code}
> The completed queries, however, do not distinguish between the outcomes of
> the completed queries.
> The proposal is to also provide success, failed, cancelled and timeout (yet
> to implement) states.
> {code:xml}
> "metrics:name=drill.queries" : {
> "completed" : {
> "successful": INTEGER,
> "failed": INTEGER,
> "cancelled": INTEGER,
> "timeout": INTEGER
> },
> "running" : INTEGER,
> "enqueued" : {
> "small" : INTEGER,
> "large" : INTEGER
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (DRILL-6328) Consolidate developer docs in docs/ folder of drill repo.
[ https://issues.apache.org/jira/browse/DRILL-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra reopened DRILL-6328: -- Closed by mistake. Reopening since it is not merged in yet. > Consolidate developer docs in docs/ folder of drill repo. > - > > Key: DRILL-6328 > URL: https://issues.apache.org/jira/browse/DRILL-6328 > Project: Apache Drill > Issue Type: Task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.14.0 > > > *For documentation* > In https://drill.apache.org/docs/developer-information/ we can add note about > docs folder with useful information for developers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6329) TPC-DS Query 66 failed due to OOM
[
https://issues.apache.org/jira/browse/DRILL-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448510#comment-16448510
]
Khurram Faraaz commented on DRILL-6329:
---
With both the options set TPC-DS query 66 still fails, due to one or more nodes
running out of memory.
Apache Drill 1.14.0 commit : 931b43e; SF1 parquet data on 4 nodes;
alter system set `drill.exec.hashagg.fallback.enabled`=true;
alter system set `planner.memory.max_query_memory_per_node` = 10737418240;
Stack trace for TPC-DS Query 66
{noformat}
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the
query.
Too little memory available
Fragment 2:0
[Error Id: 7d2abddb-eda9-4dc0-90e5-d5486942813e on qa102-45.qa.lab:31010]
(state=,code=0)
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory
while executing the query.
Too little memory available
Fragment 2:0
[Error Id: 7d2abddb-eda9-4dc0-90e5-d5486942813e on qa102-45.qa.lab:31010]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> TPC-DS Query 66 failed due to OOM
> -
>
> Key: DRILL-6329
> URL: https://issues.apache.org/jira/browse/DRILL-6329
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.14.0
>Reporter: Khurram Faraaz
>
[jira] [Commented] (DRILL-6292) Improve error message when NTLM token is sent to server using SPNEGO
[
https://issues.apache.org/jira/browse/DRILL-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448476#comment-16448476
]
Krystal commented on DRILL-6292:
We can check that if the header of the response starts with "Negotiate TIR"
then it's doing spnego over NTLM instead of over Kerberos.
> Improve error message when NTLM token is sent to server using SPNEGO
>
>
> Key: DRILL-6292
> URL: https://issues.apache.org/jira/browse/DRILL-6292
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - Java
>Reporter: Krystal
>Priority: Major
>
> The following error message is logged in drillbit.log file when the brower
> sends NTLM token instead of SPNEGO token.
> {color:#00}WARN o.a.d.e.s.r.a.DrillSpnegoLoginService - Caught
> GSSException trying to authenticate the client org.ietf.jgss.GSSException:
> Defective token detected (Mechanism level: GSSHeader did not find the right
> tag){color}
> Should have a clearer error that indicates the token sent is of NTLM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6302) NPE in Drillbit.java in close method
[
https://issues.apache.org/jira/browse/DRILL-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6302:
-
Labels: ready-to-commit (was: )
> NPE in Drillbit.java in close method
>
>
> Key: DRILL-6302
> URL: https://issues.apache.org/jira/browse/DRILL-6302
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
> Environment: git.commit.id=bb07ebbb9ba8742f44689f8bd8efb5853c5edea0
>Reporter: Khurram Faraaz
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> registrationHandle in the close method of Drillbit.java (line 228) is null,
> which causes an NPE and as a result the server does not close.
> [https://github.com/mapr/private-drill/blob/drill-1.12.0-mapr/exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java#L228]
> registrationHandle = coord.update(registrationHandle, State.QUIESCENT);
> Stack trace from drillbit.log
> {noformat}
> /opt/mapr/drill/drill-1.12.0/logs/drillbit.out
> {noformat}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
> was removed in 8.0
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:223)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
> {noformat}{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6302) NPE in Drillbit.java in close method
[
https://issues.apache.org/jira/browse/DRILL-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448433#comment-16448433
]
ASF GitHub Bot commented on DRILL-6302:
---
Github user sohami commented on the issue:
https://github.com/apache/drill/pull/1217
+1 LGTM.
> NPE in Drillbit.java in close method
>
>
> Key: DRILL-6302
> URL: https://issues.apache.org/jira/browse/DRILL-6302
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
> Environment: git.commit.id=bb07ebbb9ba8742f44689f8bd8efb5853c5edea0
>Reporter: Khurram Faraaz
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> registrationHandle in the close method of Drillbit.java (line 228) is null,
> which causes an NPE and as a result the server does not close.
> [https://github.com/mapr/private-drill/blob/drill-1.12.0-mapr/exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java#L228]
> registrationHandle = coord.update(registrationHandle, State.QUIESCENT);
> Stack trace from drillbit.log
> {noformat}
> /opt/mapr/drill/drill-1.12.0/logs/drillbit.out
> {noformat}
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
> was removed in 8.0
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:223)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
> {noformat}{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6328) Consolidate developer docs in docs/ folder of drill repo.
[ https://issues.apache.org/jira/browse/DRILL-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6328: Description: *For documentation* In https://drill.apache.org/docs/developer-information/ we can add note about docs folder with useful information for developers. > Consolidate developer docs in docs/ folder of drill repo. > - > > Key: DRILL-6328 > URL: https://issues.apache.org/jira/browse/DRILL-6328 > Project: Apache Drill > Issue Type: Task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.14.0 > > > *For documentation* > In https://drill.apache.org/docs/developer-information/ we can add note about > docs folder with useful information for developers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6328) Consolidate developer docs in docs/ folder of drill repo.
[ https://issues.apache.org/jira/browse/DRILL-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448379#comment-16448379 ] ASF GitHub Bot commented on DRILL-6328: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1220 LGTM. > Consolidate developer docs in docs/ folder of drill repo. > - > > Key: DRILL-6328 > URL: https://issues.apache.org/jira/browse/DRILL-6328 > Project: Apache Drill > Issue Type: Task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6328) Consolidate developer docs in docs/ folder of drill repo.
[ https://issues.apache.org/jira/browse/DRILL-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6328: Labels: doc-impacting ready-to-commit (was: ready-to-commit) > Consolidate developer docs in docs/ folder of drill repo. > - > > Key: DRILL-6328 > URL: https://issues.apache.org/jira/browse/DRILL-6328 > Project: Apache Drill > Issue Type: Task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6328) Consolidate developer docs in docs/ folder of drill repo.
[ https://issues.apache.org/jira/browse/DRILL-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6328: Labels: ready-to-commit (was: ) > Consolidate developer docs in docs/ folder of drill repo. > - > > Key: DRILL-6328 > URL: https://issues.apache.org/jira/browse/DRILL-6328 > Project: Apache Drill > Issue Type: Task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6328) Consolidate developer docs in docs/ folder of drill repo.
[ https://issues.apache.org/jira/browse/DRILL-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448371#comment-16448371 ] ASF GitHub Bot commented on DRILL-6328: --- Github user ilooner commented on the issue: https://github.com/apache/drill/pull/1220 replaced with TBA as suggested. > Consolidate developer docs in docs/ folder of drill repo. > - > > Key: DRILL-6328 > URL: https://issues.apache.org/jira/browse/DRILL-6328 > Project: Apache Drill > Issue Type: Task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6339) New option to disable TopN (for testing Sort)
[ https://issues.apache.org/jira/browse/DRILL-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448204#comment-16448204 ] ASF GitHub Bot commented on DRILL-6339: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1219 > New option to disable TopN (for testing Sort) > - > > Key: DRILL-6339 > URL: https://issues.apache.org/jira/browse/DRILL-6339 > Project: Apache Drill > Issue Type: Improvement > Components: Query Planning & Optimization >Affects Versions: 1.13.0 >Reporter: Boaz Ben-Zvi >Assignee: Boaz Ben-Zvi >Priority: Minor > Labels: doc-impacting, ready-to-commit > Fix For: 1.14.0 > > > When a query uses ... ORDER BY ... LIMIT ..., the planner unconditionally > picks the TopN operator to (efficiently) implement this order. This > precludes an easy way to test the External Sort operator over a large dataset > (e.g., test spilling). > A new internal option to disable picking TopN (hence using the External > Sort instead) would be useful for various testings. (And may be in other > scenarios, like a bug found in the TopN). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6334) Code cleanup
[ https://issues.apache.org/jira/browse/DRILL-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448202#comment-16448202 ] ASF GitHub Bot commented on DRILL-6334: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1213 > Code cleanup > > > Key: DRILL-6334 > URL: https://issues.apache.org/jira/browse/DRILL-6334 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Trivial > Labels: ready-to-commit > Fix For: 1.14.0 > > > Minor cleanup of a few files. Done as a separate PR to allow easier review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6289) Cluster view should show more relevant information
[ https://issues.apache.org/jira/browse/DRILL-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448201#comment-16448201 ] ASF GitHub Bot commented on DRILL-6289: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1203 > Cluster view should show more relevant information > -- > > Key: DRILL-6289 > URL: https://issues.apache.org/jira/browse/DRILL-6289 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.13.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > When fixing DRILL-6224, I noticed that the same information can be very > useful to have in the cluster view shown on a Drillbit's homepage. > The proposal is to show the following: > # Heap Memory in use > # Direct Memory (actively) in use - Since we're not able to get the total > memory held by Netty at the moment, but only what is currently allocated to > running queries > # Process CPU > # Average (System) Load Factor > Information such as the port numbers don't help much during general cluster > health, so it might be worth removing this information if more real-estate is > needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6338) License headers are not added to generated proto buf files with new license changes.
[ https://issues.apache.org/jira/browse/DRILL-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448203#comment-16448203 ] ASF GitHub Bot commented on DRILL-6338: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1215 > License headers are not added to generated proto buf files with new license > changes. > > > Key: DRILL-6338 > URL: https://issues.apache.org/jira/browse/DRILL-6338 > Project: Apache Drill > Issue Type: Bug >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Fix broken travis script -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6324) Unnest - Initial Implementation
[ https://issues.apache.org/jira/browse/DRILL-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448205#comment-16448205 ] ASF GitHub Bot commented on DRILL-6324: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1223 > Unnest - Initial Implementation > --- > > Key: DRILL-6324 > URL: https://issues.apache.org/jira/browse/DRILL-6324 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Priority: Major > Labels: ready-to-commit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6333) Generate and host Javadocs on Apache Drill website
[ https://issues.apache.org/jira/browse/DRILL-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448200#comment-16448200 ] ASF GitHub Bot commented on DRILL-6333: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1229 > Generate and host Javadocs on Apache Drill website > -- > > Key: DRILL-6333 > URL: https://issues.apache.org/jira/browse/DRILL-6333 > Project: Apache Drill > Issue Type: Improvement >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Labels: ready-to-commit > Fix For: 1.14.0 > > > Currently, there is no hosted location of Javadocs for Apache Drill. > At a minimum, there should be a cleanly generated Javadoc published for each > release, along with any Javadoc additions/enhancements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[ https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448171#comment-16448171 ] ASF GitHub Bot commented on DRILL-6281: --- Github user vrozov commented on the issue: https://github.com/apache/drill/pull/1238 DRILL-6281 is a subtask (preparation step) for DRILL-5908 Regression: Query intermittently may fail with error "Waited for 15000ms, but tasks for 'Get block maps' are not complete." Refactoring is necessary to do RCA. > Refactor TimedRunnable > -- > > Key: DRILL-6281 > URL: https://issues.apache.org/jira/browse/DRILL-6281 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[
https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448163#comment-16448163
]
ASF GitHub Bot commented on DRILL-6281:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1238#discussion_r183397526
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.drill.common.exceptions.UserException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * Class used to allow parallel executions of tasks in a simplified way.
Also maintains and reports timings of task completion.
+ * TODO: look at switching to fork join.
+ * @param The time value that will be returned when the task is
executed.
+ */
+public abstract class TimedCallable implements Callable {
+ private static final Logger logger =
LoggerFactory.getLogger(TimedCallable.class);
+
+ private static long TIMEOUT_PER_RUNNABLE_IN_MSECS = 15000;
+
+ private volatile long startTime = 0;
+ private volatile long executionTime = -1;
+
+ private static class FutureMapper implements Function, V> {
+int count;
+Throwable throwable = null;
+
+private void setThrowable(Throwable t) {
+ if (throwable == null) {
+throwable = t;
+ } else {
+throwable.addSuppressed(t);
+ }
+}
+
+@Override
+public V apply(Future future) {
+ Preconditions.checkState(future.isDone());
+ if (!future.isCancelled()) {
+try {
+ count++;
+ return future.get();
+} catch (InterruptedException e) {
+ // there is no wait as we are getting result from the
completed/done future
+ logger.error("Unexpected exception", e);
+ throw UserException.internalError(e)
+ .message("Unexpected exception")
+ .build(logger);
+} catch (ExecutionException e) {
+ setThrowable(e.getCause());
+}
+ } else {
+setThrowable(new CancellationException());
+ }
+ return null;
+}
+ }
+
+ private static class Statistics implements Consumer>
{
+final long start = System.nanoTime();
+final Stopwatch watch = Stopwatch.createStarted();
+long totalExecution = 0;
+long maxExecution = 0;
+int startedCount = 0;
+private int doneCount = 0;
+// measure thread creation times
+long earliestStart = Long.MAX_VALUE;
+long latestStart = 0;
+long totalStart = 0;
+
+@Override
+public void accept(TimedCallable task) {
+ long threadStart = task.getStartTime(TimeUnit.NANOSECONDS) - start;
+ if (threadStart >= 0) {
+startedCount++;
+earliestStart = Math.min(earliestStart, threadStart);
+latestStart = Math.max(latestStart, threadStart);
+totalS
[jira] [Commented] (DRILL-6347) Inconsistent method name "field".
[
https://issues.apache.org/jira/browse/DRILL-6347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448099#comment-16448099
]
ASF GitHub Bot commented on DRILL-6347:
---
Github user HanumathRao commented on a diff in the pull request:
https://github.com/apache/drill/pull/1236#discussion_r183384261
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/PrelVisualizerVisitor.java
---
@@ -88,10 +88,10 @@ public void endFields() {
}
public void field(String label, boolean value) {
- field(label, Boolean.toString(value));
+ append(label, Boolean.toString(value));
}
-private void field(String label, String value) {
+private void append(String label, String value) {
--- End diff --
Thanks Bruce for the changes. As Arina pointed out, appendField would be
more meaningful in this context.
> Inconsistent method name "field".
> -
>
> Key: DRILL-6347
> URL: https://issues.apache.org/jira/browse/DRILL-6347
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Assignee: KuiLIU
>Priority: Major
> Fix For: 1.14.0
>
>
> The following method is names as "field", but the method is mainly doing
> appending. So that, rename the method as "append" should be better.
> {code:java}
> private void field(String label, String value) {
> indent();
> out.append(label)
> .append(" = ")
> .append(value)
> .append("\n");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4552) During inference, treat decimal literals as Double when planner.enable_decimal_data_type is off
[
https://issues.apache.org/jira/browse/DRILL-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447938#comment-16447938
]
Volodymyr Vysotskyi commented on DRILL-4552:
In DRILL-6094 made several improvements connected with decimal data types and
with those changes, decimal literals without a cast to decimal are treated as
double literals when {{planner.enable_decimal_data_type}} is off.
In the case, when cast to decimal is used and
{{planner.enable_decimal_data_type}} is off the error is thrown.
I think this is the correct behaviour to throw an error in this case, since a
user may expect that decimal is used, but double value will be used.
> During inference, treat decimal literals as Double when
> planner.enable_decimal_data_type is off
> ---
>
> Key: DRILL-4552
> URL: https://issues.apache.org/jira/browse/DRILL-4552
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Major
>
> In SQL standard, decimal literals (e.g., 1.2, 2.5, etc) are Decimal type. In
> Drill, when `planner.enable_decimal_data_type` is off, they are treated as
> Double.
> However, the current inference mechanism is "not to do any inference if the
> operand is Decimal type". (To be more exact, treat them as ANY type and defer
> to the execution to make the decision.)
> The mechanism can be improved by treating decimal literals as Double when
> `planner.enable_decimal_data_type` is off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447905#comment-16447905
]
ASF GitHub Bot commented on DRILL-6336:
---
Github user BruceKuiLiu commented on the issue:
https://github.com/apache/drill/pull/1235
Thanks.
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6094) Decimal data type enhancements
[ https://issues.apache.org/jira/browse/DRILL-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6094: Labels: doc-impacting (was: ) > Decimal data type enhancements > -- > > Key: DRILL-6094 > URL: https://issues.apache.org/jira/browse/DRILL-6094 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: doc-impacting > Fix For: 1.14.0 > > > Currently, Decimal types are disabled by default since existing Decimal > implementation has a lot of flaws and performance problems. The goal of this > Jira to describe majority of them and possible ways of improving existing > implementation to be able to enable Decimal data types by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6094) Decimal data type enhancements
[ https://issues.apache.org/jira/browse/DRILL-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6094: Fix Version/s: 1.14.0 > Decimal data type enhancements > -- > > Key: DRILL-6094 > URL: https://issues.apache.org/jira/browse/DRILL-6094 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: doc-impacting > Fix For: 1.14.0 > > > Currently, Decimal types are disabled by default since existing Decimal > implementation has a lot of flaws and performance problems. The goal of this > Jira to describe majority of them and possible ways of improving existing > implementation to be able to enable Decimal data types by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6094) Decimal data type enhancements
[ https://issues.apache.org/jira/browse/DRILL-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6094: Reviewer: Arina Ielchiieva > Decimal data type enhancements > -- > > Key: DRILL-6094 > URL: https://issues.apache.org/jira/browse/DRILL-6094 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Volodymyr Vysotskyi >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: doc-impacting > Fix For: 1.14.0 > > > Currently, Decimal types are disabled by default since existing Decimal > implementation has a lot of flaws and performance problems. The goal of this > Jira to describe majority of them and possible ways of improving existing > implementation to be able to enable Decimal data types by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6347) Inconsistent method name "field".
[
https://issues.apache.org/jira/browse/DRILL-6347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447893#comment-16447893
]
ASF GitHub Bot commented on DRILL-6347:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1236#discussion_r183341109
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/PrelVisualizerVisitor.java
---
@@ -88,10 +88,10 @@ public void endFields() {
}
public void field(String label, boolean value) {
- field(label, Boolean.toString(value));
+ append(label, Boolean.toString(value));
}
-private void field(String label, String value) {
+private void append(String label, String value) {
--- End diff --
`appendField`?
> Inconsistent method name "field".
> -
>
> Key: DRILL-6347
> URL: https://issues.apache.org/jira/browse/DRILL-6347
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Assignee: KuiLIU
>Priority: Major
> Fix For: 1.14.0
>
>
> The following method is names as "field", but the method is mainly doing
> appending. So that, rename the method as "append" should be better.
> {code:java}
> private void field(String label, String value) {
> indent();
> out.append(label)
> .append(" = ")
> .append(value)
> .append("\n");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447884#comment-16447884
]
ASF GitHub Bot commented on DRILL-6336:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1235
@BruceKuiLiu it is not public interface, no need to deprecate the method.
You can simply replace.
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[ https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447846#comment-16447846 ] ASF GitHub Bot commented on DRILL-6281: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1238 @vrozov could you please post some line about the changes being done during refactoring, reasons and benefits? > Refactor TimedRunnable > -- > > Key: DRILL-6281 > URL: https://issues.apache.org/jira/browse/DRILL-6281 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6281) Refactor TimedRunnable
[
https://issues.apache.org/jira/browse/DRILL-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447843#comment-16447843
]
ASF GitHub Bot commented on DRILL-6281:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1238#discussion_r183330705
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java ---
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.drill.common.exceptions.UserException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import com.google.common.util.concurrent.MoreExecutors;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * Class used to allow parallel executions of tasks in a simplified way.
Also maintains and reports timings of task completion.
+ * TODO: look at switching to fork join.
+ * @param The time value that will be returned when the task is
executed.
+ */
+public abstract class TimedCallable implements Callable {
+ private static final Logger logger =
LoggerFactory.getLogger(TimedCallable.class);
+
+ private static long TIMEOUT_PER_RUNNABLE_IN_MSECS = 15000;
+
+ private volatile long startTime = 0;
+ private volatile long executionTime = -1;
+
+ private static class FutureMapper implements Function, V> {
+int count;
+Throwable throwable = null;
+
+private void setThrowable(Throwable t) {
+ if (throwable == null) {
+throwable = t;
+ } else {
+throwable.addSuppressed(t);
+ }
+}
+
+@Override
+public V apply(Future future) {
+ Preconditions.checkState(future.isDone());
+ if (!future.isCancelled()) {
+try {
+ count++;
+ return future.get();
+} catch (InterruptedException e) {
+ // there is no wait as we are getting result from the
completed/done future
+ logger.error("Unexpected exception", e);
+ throw UserException.internalError(e)
+ .message("Unexpected exception")
+ .build(logger);
+} catch (ExecutionException e) {
+ setThrowable(e.getCause());
+}
+ } else {
+setThrowable(new CancellationException());
+ }
+ return null;
+}
+ }
+
+ private static class Statistics implements Consumer>
{
+final long start = System.nanoTime();
+final Stopwatch watch = Stopwatch.createStarted();
+long totalExecution = 0;
+long maxExecution = 0;
+int startedCount = 0;
+private int doneCount = 0;
+// measure thread creation times
+long earliestStart = Long.MAX_VALUE;
+long latestStart = 0;
+long totalStart = 0;
+
+@Override
+public void accept(TimedCallable task) {
+ long threadStart = task.getStartTime(TimeUnit.NANOSECONDS) - start;
+ if (threadStart >= 0) {
+startedCount++;
+earliestStart = Math.min(earliestStart, threadStart);
+latestStart = Math.max(latestStart, threadStart);
+
[jira] [Commented] (DRILL-6272) Remove binary jars files from source distribution
[ https://issues.apache.org/jira/browse/DRILL-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447836#comment-16447836 ] ASF GitHub Bot commented on DRILL-6272: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1225 @vrozov I'll re-work PR with maven embedder, thanks for the idea. I'll ping you when changes are done. > Remove binary jars files from source distribution > - > > Key: DRILL-6272 > URL: https://issues.apache.org/jira/browse/DRILL-6272 > Project: Apache Drill > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Arina Ielchiieva >Priority: Critical > Fix For: 1.14.0 > > > Per [~vrozov] the source distribution contains binary jar files under > exec/java-exec/src/test/resources/jars -- This message was sent by Atlassian JIRA (v7.6.3#76005)
