[jira] [Created] (DRILL-6613) Refactor MaterializedField
Vlad Rozov created DRILL-6613:
-
Summary: Refactor MaterializedField
Key: DRILL-6613
URL: https://issues.apache.org/jira/browse/DRILL-6613
Project: Apache Drill
Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov
{{MaterializedField}} does not need to implement {{clone()}} and should use
constructor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6554) Minor code improvements in parquet statistics handling
Vlad Rozov created DRILL-6554: - Summary: Minor code improvements in parquet statistics handling Key: DRILL-6554 URL: https://issues.apache.org/jira/browse/DRILL-6554 Project: Apache Drill Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Avoid setting statistics in Metadata when min and max are not defined and few other minor code improvements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6490) Fix tests that rely on wrong statistics and were disabled after parquet library upgrade
Vlad Rozov created DRILL-6490: - Summary: Fix tests that rely on wrong statistics and were disabled after parquet library upgrade Key: DRILL-6490 URL: https://issues.apache.org/jira/browse/DRILL-6490 Project: Apache Drill Issue Type: Task Reporter: Vlad Rozov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6416) Unit test TestTpchDistributedConcurrent.testConcurrentQueries fails with AssertionError
[
https://issues.apache.org/jira/browse/DRILL-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved DRILL-6416.
---
Resolution: Fixed
Fix Version/s: 1.14.0
> Unit test TestTpchDistributedConcurrent.testConcurrentQueries fails with
> AssertionError
> ---
>
> Key: DRILL-6416
> URL: https://issues.apache.org/jira/browse/DRILL-6416
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build Test
>Reporter: Abhishek Girish
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> {code}
> Running org.apache.drill.TestTpchDistributedConcurrent#testConcurrentQueries
> 16:38:21.784 [2505e212-b165-7812-5c91-0a407a213964:frag:3:1] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: AssertionError
> Fragment 3:1
> [Error Id: 436120b6-5255-437e-af53-313e1c3207e0 on drillu1.qa.lab:31064]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError
> Fragment 3:1
> [Error Id: 436120b6-5255-437e-af53-313e1c3207e0 on drillu1.qa.lab:31064]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:359)
> [classes/:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:214)
> [classes/:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:325)
> [classes/:na]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.RuntimeException: java.lang.AssertionError
> at
> org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:101)
> ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:471)
> [classes/:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:313)
> [classes/:na]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: null
> at
> org.apache.drill.exec.compile.sig.MappingSet.enterConstant(MappingSet.java:85)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitBooleanConstant(EvaluationVisitor.java:1376)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitBooleanConstant(EvaluationVisitor.java:1043)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitBooleanConstant(EvaluationVisitor.java:843)
> ~[classes/:na]
> at
> org.apache.drill.common.expression.ValueExpressions$BooleanExpression.accept(ValueExpressions.java:186)
> ~[drill-logical-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitReturnValueExpression(EvaluationVisitor.java:579)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitUnknown(EvaluationVisitor.java:342)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitUnknown(EvaluationVisitor.java:1399)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:1084)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:843)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.filter.ReturnValueExpression.accept(ReturnValueExpression.java:56)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.EvaluationVisitor.addExpr(EvaluationVisitor.java:100)
> ~[classes/:na]
> at
> org.apache.drill.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:334)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.setupWorker(NestedLoopJoinBatch.java:266)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.buildSchema(NestedLoopJoinBatch.java:384)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:229)
>
[jira] [Created] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
Vlad Rozov created DRILL-6435:
-
Summary: MappingSet is stateful, so it can't be shared between
threads
Key: DRILL-6435
URL: https://issues.apache.org/jira/browse/DRILL-6435
Project: Apache Drill
Issue Type: Bug
Reporter: Vlad Rozov
Assignee: Vlad Rozov
There are several instances where static {{MappingSet}} instances are used (for
example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes instance
reuse across threads when queries are executed concurrently. As {{MappingSet}}
is a stateful class with visitor design pattern, such reuse causes invalid
state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6425) Upgrade org.ojai:ojai version
Vlad Rozov created DRILL-6425:
-
Summary: Upgrade org.ojai:ojai version
Key: DRILL-6425
URL: https://issues.apache.org/jira/browse/DRILL-6425
Project: Apache Drill
Issue Type: Task
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6353) Upgrade Parquet MR dependencies
Vlad Rozov created DRILL-6353:
-
Summary: Upgrade Parquet MR dependencies
Key: DRILL-6353
URL: https://issues.apache.org/jira/browse/DRILL-6353
Project: Apache Drill
Issue Type: Task
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DRILL-6252) Foreman node is going down when the non foreman node is stopped
[
https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved DRILL-6252.
---
Resolution: Fixed
Fixed as part of DRILL-6295
> Foreman node is going down when the non foreman node is stopped
> ---
>
> Key: DRILL-6252
> URL: https://issues.apache.org/jira/browse/DRILL-6252
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Venkata Jyothsna Donapati
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: foreman_drillbit.log, nonforeman_drillbit.log
>
>
> Two drillbits are running. I'm running a join query over parquet and tried to
> stop the non-foreman node using drillbit.sh stop. The query fails with
> *"Error: DATA_READ ERROR: Exception occurred while reading from disk".* The
> non-foreman node goes down. The foreman node also goes down. When I looked at
> the drillbit.log of both foreman and non-foreman I found that there is memory
> leak "Memory was leaked by query. Memory leaked:
> (2097152)\nAllocator(op:2:0:0:HashPartitionSender)
> 100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are
> the stack traces for memory leaks
> {noformat}
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100
> (res/actual/peak/limit)
>
>
> Fragment 2:1
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100
> (res/actual/peak/limit)
> {noformat}
>
> Ping me for the logs and more information.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6295) {{PartitionerDecorator}} may close {{partitioners}} while {{CustomRunnable}} are active during query cancellation
Vlad Rozov created DRILL-6295:
-
Summary: {{PartitionerDecorator}} may close {{partitioners}} while
{{CustomRunnable}} are active during query cancellation
Key: DRILL-6295
URL: https://issues.apache.org/jira/browse/DRILL-6295
Project: Apache Drill
Issue Type: Bug
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Fix For: 1.14.0
During query cancellation, in case
{{PartitionerDecorator.executeMethodLogic()}} is active (waiting on the
{{latch}}), the wait will be interrupted and {{Future}}s cancelled, but there
is no guarantee that all {{CustomRunnable}} terminate before returning from
{{PartitionerDecorator.executeMethodLogic()}}. On exit, both income and
outgoing batches are cleared, leading to clearing of underlying {{Vector}}s and
{{DrillBuf}}s. This eventually causes unallocated memory access and JVM crash
as {{CustomRunnable}} may execute after income/outgoing batches are cleared.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6288) Upgrade org.javassist:javassist and org.reflections:reflections
Vlad Rozov created DRILL-6288:
-
Summary: Upgrade org.javassist:javassist and
org.reflections:reflections
Key: DRILL-6288
URL: https://issues.apache.org/jira/browse/DRILL-6288
Project: Apache Drill
Issue Type: Task
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Fix For: 1.14.0
Current {{org.javassist:javassist}} version {{3.16.1-GA}} does not support JDK
1.8. Need to upgrade to {{3.18.2-GA}} or above. See [Reflections - Java 8 -
invalid constant type
|https://stackoverflow.com/questions/30313255/reflections-java-8-invalid-constant-type]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6287) apache-release profile should be disabled by default
Vlad Rozov created DRILL-6287: - Summary: apache-release profile should be disabled by default Key: DRILL-6287 URL: https://issues.apache.org/jira/browse/DRILL-6287 Project: Apache Drill Issue Type: Bug Reporter: Vlad Rozov Assignee: Vlad Rozov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6286) Regression: incorrect reference to shutdown in drillbit.log
Vlad Rozov created DRILL-6286:
-
Summary: Regression: incorrect reference to shutdown in
drillbit.log
Key: DRILL-6286
URL: https://issues.apache.org/jira/browse/DRILL-6286
Project: Apache Drill
Issue Type: Bug
Reporter: Vlad Rozov
Assignee: Timothy Farkas
drillbit.log refers to shutdown even in cases when no shutdown sequence was
initiated:
{noformat}
2018-03-16 11:55:52,693 [drill-executor-19] INFO
o.apache.drill.exec.work.WorkManager - Waiting for 0 queries to complete before
shutting down
2018-03-16 11:55:52,693 [drill-executor-19] INFO
o.apache.drill.exec.work.WorkManager - Waiting for 3 running fragments to
complete before shutting down
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6281) Refactor TimedRunnable
Vlad Rozov created DRILL-6281: - Summary: Refactor TimedRunnable Key: DRILL-6281 URL: https://issues.apache.org/jira/browse/DRILL-6281 Project: Apache Drill Issue Type: Sub-task Reporter: Vlad Rozov Assignee: Vlad Rozov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6280) Cleanup execution of BuildTimeScan during maven build
Vlad Rozov created DRILL-6280:
-
Summary: Cleanup execution of BuildTimeScan during maven build
Key: DRILL-6280
URL: https://issues.apache.org/jira/browse/DRILL-6280
Project: Apache Drill
Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov
{{exec:java}} requires {{test}} scope due to dependency on {{logback-classic}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6273) Remove dependency licensed under Category X
Vlad Rozov created DRILL-6273: - Summary: Remove dependency licensed under Category X Key: DRILL-6273 URL: https://issues.apache.org/jira/browse/DRILL-6273 Project: Apache Drill Issue Type: Task Reporter: Vlad Rozov Fix For: 1.14.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6272) Remove binary jars files from source distribution
Vlad Rozov created DRILL-6272: - Summary: Remove binary jars files from source distribution Key: DRILL-6272 URL: https://issues.apache.org/jira/browse/DRILL-6272 Project: Apache Drill Issue Type: Task Reporter: Vlad Rozov Fix For: 1.14.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6271) Update copyright range in NOTICE
Vlad Rozov created DRILL-6271: - Summary: Update copyright range in NOTICE Key: DRILL-6271 URL: https://issues.apache.org/jira/browse/DRILL-6271 Project: Apache Drill Issue Type: Task Reporter: Vlad Rozov Fix For: 1.14.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6237) Upgrade checkstyle version to 5.9 or above
Vlad Rozov created DRILL-6237: - Summary: Upgrade checkstyle version to 5.9 or above Key: DRILL-6237 URL: https://issues.apache.org/jira/browse/DRILL-6237 Project: Apache Drill Issue Type: Task Reporter: Vlad Rozov Assignee: Vlad Rozov Checkstyle versions prior to 5.9 do not support Java 8 syntax. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6202) Deprecate usage of IndexOutOfBoundsException to re-alloc vectors
Vlad Rozov created DRILL-6202: - Summary: Deprecate usage of IndexOutOfBoundsException to re-alloc vectors Key: DRILL-6202 URL: https://issues.apache.org/jira/browse/DRILL-6202 Project: Apache Drill Issue Type: Bug Reporter: Vlad Rozov Assignee: Vlad Rozov As bounds checking may be enabled or disabled, using IndexOutOfBoundsException to resize vectors is unreliable. It works only when bounds checking is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6164) Heap memory leak during parquet scan and OOM
Vlad Rozov created DRILL-6164: - Summary: Heap memory leak during parquet scan and OOM Key: DRILL-6164 URL: https://issues.apache.org/jira/browse/DRILL-6164 Project: Apache Drill Issue Type: Bug Reporter: Vlad Rozov Assignee: Vlad Rozov During a scan of a large set of parquet files, Drill iterates over the set initializing parquet readers. Such initialization may require a significant memory usage (both heap and direct). When scan moves to the next parquet file in the set, it does not remove reference to the reader from the set it iterates over and does not remove references created during initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6152) For CSV and TSV files splitting should be disabled in case of quoted processing
Vlad Rozov created DRILL-6152: - Summary: For CSV and TSV files splitting should be disabled in case of quoted processing Key: DRILL-6152 URL: https://issues.apache.org/jira/browse/DRILL-6152 Project: Apache Drill Issue Type: Bug Reporter: Vlad Rozov With quoted processing enabled it is not possible to process csv or tsv file from an arbitrary offset. Consider a case where a csv file is split inside a quoted string that contains a line feed. As beginning quote is part of another split, line feed would be recognized as a regular line feed instead of a line feed inside quotes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6151) Fragment executors may terminate without sending final batch to a downstream causing query to hang
Vlad Rozov created DRILL-6151: - Summary: Fragment executors may terminate without sending final batch to a downstream causing query to hang Key: DRILL-6151 URL: https://issues.apache.org/jira/browse/DRILL-6151 Project: Apache Drill Issue Type: Bug Reporter: Vlad Rozov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6150) org.apache.drill.exec.store.easy.text.compliant.TextReader is subject of java.lang.StackOverflowError
Vlad Rozov created DRILL-6150:
-
Summary:
org.apache.drill.exec.store.easy.text.compliant.TextReader is subject of
java.lang.StackOverflowError
Key: DRILL-6150
URL: https://issues.apache.org/jira/browse/DRILL-6150
Project: Apache Drill
Issue Type: Bug
Components: Storage - Text CSV
Reporter: Vlad Rozov
Attachments: test.tsv.zip
TextReader uses recursion to parse quoted values causing StackOverflowError:
{noformat}
select count(*) from dfs.`test.tsv`
{noformat}
{noformat}
Exception in thread "drill-executor-26" java.lang.StackOverflowError
at
org.apache.drill.exec.store.easy.text.compliant.TextReader.parseQuotedValue(TextReader.java:269)
at
org.apache.drill.exec.store.easy.text.compliant.TextReader.parseQuotedValue(TextReader.java:242)
at
org.apache.drill.exec.store.easy.text.compliant.TextReader.parseQuotedValue(TextReader.java:293)
at
org.apache.drill.exec.store.easy.text.compliant.TextReader.parseQuotedValue(TextReader.java:242)
at
org.apache.drill.exec.store.easy.text.compliant.TextReader.parseQuotedValue(TextReader.java:293)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6086) TestSortImpl.testLargeBatch unit test fails randomly
Vlad Rozov created DRILL-6086:
-
Summary: TestSortImpl.testLargeBatch unit test fails randomly
Key: DRILL-6086
URL: https://issues.apache.org/jira/browse/DRILL-6086
Project: Apache Drill
Issue Type: Bug
Reporter: Vlad Rozov
{noformat}
Failed tests:
TestSortImpl.testLargeBatch:513->runJumboBatchTest:486->runLargeSortTest:455
Value of 1:0 expected:<0> but was:<1>
{noformat}
The test fails due to memory corruption caused by a write out of the direct
buffer allocated space. With bounds check enabled, the test fails reliably with
{noformat}
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/bin/java
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:57731,suspend=y,server=n
-Dvisualvm.id=131461133353377 -Ddrill.exec.rpc.user.timeout=0
-Ddrill.exec.rpc.bit.timeout=0 -Dlog.path=${DRILL_LOG_DIR}/drill.log
-Dlog.query.path=${DRILL_LOG_DIR}/query.log
-Djava.io.tmpdir=/Users/vrozov/Projects/Apache/drill/exec/java-exec/target
-Xms512m -Xmx4096m -Ddrill.exec.http.enabled=false
-Ddrill.exec.sys.store.provider.local.write=false
-Dorg.apache.drill.exec.server.Drillbit.system_options=org.apache.drill.exec.compile.ClassTransformer.scalar_replacement=on
-Ddrill.test.query.printing.silent=true
-Ddrill.catastrophic_to_standard_out=true -XX:MaxPermSize=512M
-XX:MaxDirectMemorySize=3072M -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true -XX:+CMSClassUnloadingEnabled -ea
-Didea.test.cyclic.buffer.size=1048576
-javaagent:/Users/vrozov/Library/Caches/IntelliJIdea2017.3/captureAgent/debugger-agent.jar=/private/var/folders/52/11m3mlk902g_wwp856y3sdvcgp/T/capture.props
-Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ
IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ
IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ
[jira] [Created] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
Vlad Rozov created DRILL-6053: - Summary: Avoid excessive locking in LocalPersistentStore Key: DRILL-6053 URL: https://issues.apache.org/jira/browse/DRILL-6053 Project: Apache Drill Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov When query profiles are written to LocalPersistentStore, the write is unnecessary serialized due to read/write lock that was introduced for versioned PersistentStore. Only versioned access needs to be protected by read/write lock. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-6030) Managed sort should minimize number of batches in a k-way merge
Vlad Rozov created DRILL-6030: - Summary: Managed sort should minimize number of batches in a k-way merge Key: DRILL-6030 URL: https://issues.apache.org/jira/browse/DRILL-6030 Project: Apache Drill Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov The time complexity of the algorithm is O(n*k*log(k)) where k is a number of batches to merge and n is a number of records in each batch (assuming equal size batches). As n*k is the total number of record to merge and it can be quite large, minimizing k should give better results. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-6004) Direct buffer bounds checking should be disabled by default
Vlad Rozov created DRILL-6004:
-
Summary: Direct buffer bounds checking should be disabled by
default
Key: DRILL-6004
URL: https://issues.apache.org/jira/browse/DRILL-6004
Project: Apache Drill
Issue Type: Improvement
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Priority: Minor
Direct buffer bounds checking is enabled either when assertions are enabled
(see DRILL-6001) or when {{drill.enable_unsafe_memory_access}} property is not
set to true, so it is enabled in production as by default
{{drill.enable_unsafe_memory_access}} is not set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-6002) Avoid memory copy from direct buffer to heap while spilling to local disk
Vlad Rozov created DRILL-6002: - Summary: Avoid memory copy from direct buffer to heap while spilling to local disk Key: DRILL-6002 URL: https://issues.apache.org/jira/browse/DRILL-6002 Project: Apache Drill Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov When spilling to a local disk or to any file system that supports WritableByteChannel it is preferable to avoid copy from off-heap to java heap as WritableByteChannel can work directly with the off-heap memory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-6001) Deprecate using assertions (-ea) to enable direct memory allocation tracing.
Vlad Rozov created DRILL-6001: - Summary: Deprecate using assertions (-ea) to enable direct memory allocation tracing. Key: DRILL-6001 URL: https://issues.apache.org/jira/browse/DRILL-6001 Project: Apache Drill Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor Drill uses assertion (-ea) to enable memory allocation tracing. Most of the time assertions are enabled/disabled globally (for all packages) by using "-ea" java command line option and it leads to excessive CPU and heap utilization. It will be better to limit the impact of assertion enabled to the java "assert" statement as expected by a majority of Java developers and use a separate property (that already exists) to enable/disable direct memory allocation tracing/debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5961) For long running queries (> 10 min) Drill may raise FragmentSetupException for completed/cancelled fragments
Vlad Rozov created DRILL-5961:
-
Summary: For long running queries (> 10 min) Drill may raise
FragmentSetupException for completed/cancelled fragments
Key: DRILL-5961
URL: https://issues.apache.org/jira/browse/DRILL-5961
Project: Apache Drill
Issue Type: Bug
Reporter: Vlad Rozov
Assignee: Vlad Rozov
{{WorkEventBus}} uses {{recentlyFinishedFragments}} cache to check for
completed or cancelled fragments. Such check is not reliable as entries in
{{recentlyFinishedFragments}} expire after 10 minutes, so
{{FragmentSetupException}} is raised even for completed or cancelled queries.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5936) Refactor MergingRecordBatch based on code review
Vlad Rozov created DRILL-5936: - Summary: Refactor MergingRecordBatch based on code review Key: DRILL-5936 URL: https://issues.apache.org/jira/browse/DRILL-5936 Project: Apache Drill Issue Type: Improvement Components: Tools, Build & Test Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5905) Exclude jdk-tools from project dependencies
Vlad Rozov created DRILL-5905: - Summary: Exclude jdk-tools from project dependencies Key: DRILL-5905 URL: https://issues.apache.org/jira/browse/DRILL-5905 Project: Apache Drill Issue Type: Improvement Components: Tools, Build & Test Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor hadoop-annotations and hbase-annotations have system scope dependency on JDK tools.jar. This dependency is provided by JDK and should be excluded from the project dependencies -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5862) Update project parent pom xml to the latest ASF version
Vlad Rozov created DRILL-5862: - Summary: Update project parent pom xml to the latest ASF version Key: DRILL-5862 URL: https://issues.apache.org/jira/browse/DRILL-5862 Project: Apache Drill Issue Type: Improvement Components: Tools, Build & Test Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (DRILL-5813) A query that includes sort loses Drill connection. Drill sometimes crashes.
[
https://issues.apache.org/jira/browse/DRILL-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved DRILL-5813.
---
Resolution: Not A Bug
Fix Version/s: (was: 1.12.0)
When Drill is started with assertion enabled (-ea) it tracks every memory
allocation and the option requires more memory. See if the same is reproducible
with assertion disabled (the option that will be used in production).
> A query that includes sort loses Drill connection. Drill sometimes crashes.
>
>
> Key: DRILL-5813
> URL: https://issues.apache.org/jira/browse/DRILL-5813
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.11.0
>Reporter: Robert Hou
>Assignee: Vlad Rozov
> Attachments: drillbit.log, drill.log
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.enable_decimal_data_type` = true;
> select count(*) from (select * from
> dfs.`/drill/testdata/resource-manager/all_types_large` order by missing11) d
> where d.missing3 is false;
> {noformat}
> This query has passed before when the number of threads and amount of memory
> is restricted. With more threads and memory, the query does not complete
> execution.
> Here is the stack trace:
> {noformat}
> Exception occurred with closed channel. Connection: /10.10.100.190:59281
> <--> /10.10.100.190:31010 (user client)
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
> at
> oadd.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
> at oadd.io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407)
> at
> oadd.io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:32)
> at oadd.io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:792)
> at
> oadd.io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280)
> at
> oadd.io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26)
> at
> oadd.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
> at
> oadd.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
> at
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
> at
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> User Error Occurred: Connection /10.10.100.190:59281 <-->
> /10.10.100.190:31010 (user client) closed unexpectedly. Drillbit down?
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR:
> Connection /10.10.100.190:59281 <--> /10.10.100.190:31010 (user client)
> closed un
> expectedly. Drillbit down?
> [Error Id: b97704a4-b8f0-4cd0-b428-2cf1bcf39a1d ]
> at
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> at
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> at
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> at
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> at
>
[jira] [Created] (DRILL-5791) Unit test Jackson polymorphic unmarshalling
Vlad Rozov created DRILL-5791: - Summary: Unit test Jackson polymorphic unmarshalling Key: DRILL-5791 URL: https://issues.apache.org/jira/browse/DRILL-5791 Project: Apache Drill Issue Type: Test Reporter: Vlad Rozov Assignee: Vlad Rozov -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))
Vlad Rozov created DRILL-5784:
-
Summary: SYSTEM ERROR: IndexOutOfBoundsException: index: 512,
length: 4 (expected: range(0, 512))
Key: DRILL-5784
URL: https://issues.apache.org/jira/browse/DRILL-5784
Project: Apache Drill
Issue Type: Bug
Environment: planner.slice_target > 10
planner.enable_nljoin_for_scalar_only = false
Reporter: Vlad Rozov
Assignee: Vlad Rozov
The following query causes IndexOutOfBoundsException:
{code}
SELECT
`t1`.`one` `one`
FROM
(
SELECT
1 `one`
FROM
dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
INNER JOIN (
SELECT
314 `c_integer`
FROM
dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
) `t0` ON (
`/drill/exec/java-exec/src/test/resources/join/j1`.c_integer IS NOT
DISTINCT
FROM
`t0`.`c_integer`
)
GROUP BY
`one`
) `t1`
INNER JOIN (
SELECT
count(1) `measure`
FROM
dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
) `t5` ON TRUE
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
