Re: Can this scenario cause a query to hang ?

Hey John, thanks for sharing your experience. If you see this again try collecting the jstack output for the foreman node of the query, and also check in the query profile which fragments are still marked as RUNNING. Thanks On Thu, Apr 7, 2016 at 2:29 PM, John Omernik wrote:

Re: Can this scenario cause a query to hang ?

On Fri, Apr 8, 2016 at 12:14 AM, Sudheesh Katkam wrote: > I agree there is a problem. Nice catch! Is there a ticket for this? > Not yet, I wanted to confirm I didn't miss anything obvious first. Will create a JIRA right away. > The fragment executor is responsible for

Apache Drill Plan... - Delete: ";;"

George Chow added a suggestion to Apache Drill Plan Syntax (https://docs.google.com/document/d/1QTL8warUYS2KjldQrGUse7zp8eA72VKtLOHwfXy6c7I/edit?disco=Aqd0ngk=suggestion_email_discussion) George Chow Delete: ";;" You received this email because you are subscribed to all comments on

Apache Drill Plan... - Add: ";;"

inactiv 9 added a suggestion to Apache Drill Plan Syntax (https://docs.google.com/document/d/1QTL8warUYS2KjldQrGUse7zp8eA72VKtLOHwfXy6c7I/edit?disco=Aqd0ngI=suggestion_email_discussion) inactiv 9 Add: ";;" You received this email because you are subscribed to all comments on Apache

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

Github user jacques-n commented on a diff in the pull request: https://github.com/apache/drill/pull/443#discussion_r58974382 --- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java --- @@ -19,36 +19,75 @@ import

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user laurentgo commented on a diff in the pull request: https://github.com/apache/drill/pull/430#discussion_r58969685 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/MurmurHash3.java --- @@ -0,0 +1,280 @@ +/** + * Licensed to the Apache

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user amansinha100 commented on the pull request: https://github.com/apache/drill/pull/430#issuecomment-207153608 @jacques-n regarding re-implementation, agree with the general concern; however in this case the core logic is directly taken from the java based murmur hash

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user laurentgo commented on a diff in the pull request: https://github.com/apache/drill/pull/430#discussion_r58968289 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/XXHash.java --- @@ -24,14 +24,15 @@ import

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user laurentgo commented on a diff in the pull request: https://github.com/apache/drill/pull/430#discussion_r58968083 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/TestHashBase.java --- @@ -0,0 +1,72 @@ +/** + * Licensed to the Apache Software

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user laurentgo commented on a diff in the pull request: https://github.com/apache/drill/pull/430#discussion_r58967457 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DrillHash.java --- @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user laurentgo commented on a diff in the pull request: https://github.com/apache/drill/pull/430#discussion_r58967081 --- Diff: common/src/main/java/org/apache/drill/common/util/DrillStringUtils.java --- @@ -163,10 +163,10 @@ private static void appendByte(StringBuilder

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/430#issuecomment-207147567 Just to clarify since I didn't say so. I'm ok with this to be merged given Aman's +1. --- If your project is set up for it, you can reply to this email and have your

[GitHub] drill pull request: DRILL-4237 DRILL-4478 fully implement hash to ...

Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/430#issuecomment-207147616 Did you guys do a perf analysis to ensure no regressions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

Re: Can this scenario cause a query to hang ?

I agree there is a problem. Nice catch! Is there a ticket for this? The fragment executor is responsible for sending the final state, and in this case its waiting forever, making the query hang. In any scenario where a thread other than the fragment executor is failing (or cancelling) a fragment,

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user amansinha100 commented on the pull request: https://github.com/apache/drill/pull/468#issuecomment-207132298 LGTM. +1. Since this fix is not directly changing the cost of per-row filter evaluation itself (it is reducing the number of rows on which filter evaluation is

[GitHub] drill pull request: DRILL-2100: Added deleting temporary spill dir...

Github user vdiravka commented on the pull request: https://github.com/apache/drill/pull/454#issuecomment-207131838 @jaltekruse Could you please review this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] drill pull request: DRILL-4459: SchemaChangeException while queryi...

Github user vdiravka commented on the pull request: https://github.com/apache/drill/pull/431#issuecomment-207126641 @vkorukanti I made a commit with rebasing on master and updating the commit message after your comment. Please make a quick review of this changes. --- If

Re: Can this scenario cause a query to hang ?

Thanks Sudheesh. So even after we fix DRILL-3714, it's still possible for the root fragment to fail without being cancelled. Take a look at BaseRawBatchBuffer.enqueue() and you will see that, once a fragment is in failed state, this method will release the batch and send back an OK ack to the

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58957037 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/planner/fragment/TestHardAffinityFragmentParallelizer.java --- @@ -0,0 +1,192 @@ +/**

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58956953 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/DistributionAffinity.java --- @@ -0,0 +1,63 @@ +/** + * Licensed

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58957028 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/FindLimit0Visitor.java --- @@ -43,7 +50,15 @@ public static

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58956969 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/HardAffinityFragmentParallelizer.java --- @@ -0,0 +1,134 @@ +/**

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58956963 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/HardAffinityFragmentParallelizer.java --- @@ -0,0 +1,134 @@ +/**

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58956976 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/ParallelizationParameters.java --- @@ -0,0 +1,32 @@ +/** + *

[GitHub] drill pull request: DRILL-4446: Support mandatory work assignment ...

Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/403#discussion_r58956992 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/FindLimit0Visitor.java --- @@ -116,4 +131,33 @@ public RelNode

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/468#issuecomment-207115616 @amansinha100 and @hsuanyi , I revised PR based on your comments. Can you take another look? thx. --- If your project is set up for it, you can reply to this

[jira] [Created] (DRILL-4594) Incorrect results when querying AVRO data with record and array in schema

Richard Henderson created DRILL-4594: Summary: Incorrect results when querying AVRO data with record and array in schema Key: DRILL-4594 URL: https://issues.apache.org/jira/browse/DRILL-4594

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58953148 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSDirPartitionLocation.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the

[GitHub] drill pull request: DRILL-4593: Remove OldAssignmentCreator in Fil...

GitHub user vkorukanti opened a pull request: https://github.com/apache/drill/pull/473 DRILL-4593: Remove OldAssignmentCreator in FileSystemPlugin + Remove dead code in ParquetGroupScan You can merge this pull request into a Git repository by running: $ git pull

[jira] [Created] (DRILL-4593) Remove OldAssignmentCreator in FileSystemPlugin

Venki Korukanti created DRILL-4593: -- Summary: Remove OldAssignmentCreator in FileSystemPlugin Key: DRILL-4593 URL: https://issues.apache.org/jira/browse/DRILL-4593 Project: Apache Drill

Re: Can this scenario cause a query to hang ?

I can answer one question myself. See inline. As you mentioned elsewhere, this issue will rarely happen (and even harder to reproduce) once DRILL-3714 is committed. On Thu, Apr 7, 2016 at 11:38 AM, Sudheesh Katkam wrote: > Hakim, > > Can you point me to where [3] happens?

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58949303 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSDirPartitionLocation.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user magpierre commented on the pull request: https://github.com/apache/drill/pull/451#issuecomment-207091026 I will do the requested, thanks @jaltekruse. Just a quick note on the XML files it can support: I have tested around 100 different XML formats ranging from simple to

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user magpierre commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58944903 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLFormatPlugin.java --- @@ -0,0 +1,139 @@ +/** + * Licensed to

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user magpierre commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58944565 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLRecordReader.java --- @@ -0,0 +1,95 @@ +/** + * Licensed to the

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user magpierre commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58944333 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLRecordReader.java --- @@ -0,0 +1,95 @@ +/** + * Licensed to the

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user magpierre commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58943891 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLRecordReader.java --- @@ -0,0 +1,95 @@ +/** + * Licensed to the

[GitHub] drill pull request: DRILL-4571: Add link to local Drill logs from ...

GitHub user arina-ielchiieva opened a pull request: https://github.com/apache/drill/pull/472 DRILL-4571: Add link to local Drill logs from the web UI You can merge this pull request into a Git repository by running: $ git pull https://github.com/arina-ielchiieva/drill

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58926583 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSDirPartitionLocation.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the

Re: Can this scenario cause a query to hang ?

Hakim, Can you point me to where [3] happens? Two questions: + Why is the root fragment blocked? If the user channel is closed, the query is cancelled [1], which should cancel and interrupt all running fragments. This interruption happens regardless of fragment failure that you have pointed out

[GitHub] drill pull request: DRILL-4592: Explain plan statement should show...

GitHub user jinfengni opened a pull request: https://github.com/apache/drill/pull/471 DRILL-4592: Explain plan statement should show plan in WebUI @amansinha100 , could you review this small code patch? Thanks. For now, there is lack of a way to add unit test for all WebUI

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58921685 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java --- @@ -148,13 +139,41 @@ public String

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user amansinha100 commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58921279 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSDirPartitionLocation.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the

[jira] [Created] (DRILL-4592) Explain plan statement should show plan in WebUi

Jinfeng Ni created DRILL-4592: - Summary: Explain plan statement should show plan in WebUi Key: DRILL-4592 URL: https://issues.apache.org/jira/browse/DRILL-4592 Project: Apache Drill Issue Type:

[GitHub] drill pull request: DRILL-3714: Avoid cascading disconnection when...

Github user sudheeshkatkam commented on the pull request: https://github.com/apache/drill/pull/463#issuecomment-207025128 Overall change looks good; I have some minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X

Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-207022155 @jacques-n I'll wait for your final review before I merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58912269 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSDirPartitionLocation.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the

[jira] [Resolved] (DRILL-4330) Long running SQL query hangs once Foreman node is killed

[ https://issues.apache.org/jira/browse/DRILL-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam resolved DRILL-4330. Resolution: Fixed Fixed through DRILL-3743. > Long running SQL query hangs once Foreman

[jira] [Resolved] (DRILL-4588) Enable JMXReporter to Expose Metrics

[ https://issues.apache.org/jira/browse/DRILL-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam resolved DRILL-4588. Resolution: Fixed Fixed in

[jira] [Resolved] (DRILL-3743) query hangs on sqlline once Drillbit on foreman node is killed

[ https://issues.apache.org/jira/browse/DRILL-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam resolved DRILL-3743. Resolution: Fixed Fixed in

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user jinfengni commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58911205 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java --- @@ -148,13 +139,41 @@ public String getName(int

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user jaltekruse commented on the pull request: https://github.com/apache/drill/pull/451#issuecomment-206989347 Hey @magpierre Thanks for the work on this, we have had request for an XML reader from a number of community members in the past. This is the right way to

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user jaltekruse commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58904346 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLRecordReader.java --- @@ -0,0 +1,95 @@ +/** + * Licensed to

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user jaltekruse commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58903153 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLFormatPlugin.java --- @@ -0,0 +1,139 @@ +/** + * Licensed to

[GitHub] drill pull request: Drill 3878: XML support in Apache Drill

Github user jaltekruse commented on a diff in the pull request: https://github.com/apache/drill/pull/451#discussion_r58902477 --- Diff: contrib/storage-xml/src/main/java/org/apache/drill/exec/store/easy/xml/XMLFormatPlugin.java --- @@ -0,0 +1,139 @@ +/** + * Licensed to

Contributing to Drill?

A Drill user is eager to contribute a new XML plugin, and was planning to follow the instructions noted at https://drill.apache.org/docs/apache-drill-contribution-guidelines/. However, my understanding is we're no longer following that process. Can one of the dev commiters update that Web page

[jira] [Created] (DRILL-4591) Extend config system with site, node property files

Paul Rogers created DRILL-4591: -- Summary: Extend config system with site, node property files Key: DRILL-4591 URL: https://issues.apache.org/jira/browse/DRILL-4591 Project: Apache Drill Issue

[jira] [Created] (DRILL-4590) Queries return wrong results when applied to views

Dechang Gu created DRILL-4590: - Summary: Queries return wrong results when applied to views Key: DRILL-4590 URL: https://issues.apache.org/jira/browse/DRILL-4590 Project: Apache Drill Issue

Re: Can this scenario cause a query to hang ?

Abdel - I think I've seen this on a MapR cluster I run, especially on CTAS. For me, I have not brought it up because the cluster I am running on has some serious personal issues (like being hardware that's near 7 years old, its a test cluster) and given the "hard to reproduce" nature of the

[jira] [Resolved] (DRILL-4523) Disallow using loopback address in distributed mode

[ https://issues.apache.org/jira/browse/DRILL-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-4523. - Resolution: Fixed Fixed in 9514cbe. > Disallow using loopback address in distributed

[jira] [Resolved] (DRILL-4544) Improve error messages for REFRESH TABLE METADATA command

[ https://issues.apache.org/jira/browse/DRILL-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva resolved DRILL-4544. - Resolution: Fixed Merged into master. Commit id - a759552. > Improve error messages for

[GitHub] drill pull request: DRILL-4143: Fix inconsistency in writing metad...

GitHub user chunhui-shi opened a pull request: https://github.com/apache/drill/pull/470 DRILL-4143: Fix inconsistency in writing metadata with different user's permission. When accessing readBlockMeta, use an impersonated special Filesystem object which is of drillbituser. So we

Can this scenario cause a query to hang ?

So, we've been seeing some queries hang, I've come up with a possible explanation, but so far it's really difficult to reproduce. Let me know if you think this explanation doesn't hold up or if you have any ideas how we can reproduce it. Thanks - generally it's a CTAS running on a large cluster

[GitHub] drill pull request: DRILL-3714: Avoid cascading disconnection when...

Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/463#discussion_r58826205 --- Diff: exec/rpc/src/main/java/org/apache/drill/exec/rpc/RemoteConnection.java --- @@ -121,17 +115,72 @@ public void disable() { } }

[GitHub] drill pull request: DRILL-3714: Avoid cascading disconnection when...

Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/463#discussion_r58825673 --- Diff: exec/rpc/src/main/java/org/apache/drill/exec/rpc/RequestIdMap.java --- @@ -20,51 +20,82 @@ import io.netty.buffer.ByteBuf; import

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user hsuanyi commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58824977 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/DFSDirPartitionLocation.java --- @@ -0,0 +1,70 @@ +/** + * Licensed to the

[GitHub] drill pull request: DRILL-4589: Reduce planning time for file syst...

Github user hsuanyi commented on a diff in the pull request: https://github.com/apache/drill/pull/468#discussion_r58824883 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/FileSystemPartitionDescriptor.java --- @@ -148,13 +139,41 @@ public String getName(int