[jira] [Created] (DRILL-7238) Drill does not use DirectScan for non-existent columns
Venkata Jyothsna Donapati created DRILL-7238:
Summary: Drill does not use DirectScan for non-existent columns
Key: DRILL-7238
URL: https://issues.apache.org/jira/browse/DRILL-7238
Project: Apache Drill
Issue Type: Bug
Reporter: Venkata Jyothsna Donapati
Assignee: Venkata Jyothsna Donapati
This query does not use the summary metadata cache file:
select count(int_nulls_id), count(int_id), count(ss_ticket_number),
count(extra) from store_sales_null_blocks_int;
In this query, extra is a column that does not exist (non-existent column).
Here is the explain plan:
{noformat}
| 00-00Screen
00-01 Project(EXPR$0=[$0], EXPR$1=[$1], EXPR$2=[$2], EXPR$3=[$3])
00-02StreamAgg(group=[{}], EXPR$0=[$SUM0($0)], EXPR$1=[$SUM0($1)],
EXPR$2=[$SUM0($2)], EXPR$3=[$SUM0($3)])
00-03 UnionExchange
01-01StreamAgg(group=[{}], EXPR$0=[COUNT($0)], EXPR$1=[COUNT($1)],
EXPR$2=[COUNT($2)], EXPR$3=[COUNT($3)])
01-02 Scan(table=[[dfs, parquet_metadata_cache,
store_sales_null_blocks_int]], groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath
[path=/drill/testdata/metadata_cache/store_sales_null_blocks_int]],
selectionRoot=/drill/testdata/metadata_cache/store_sales_null_blocks_int,
numFiles=1, numRowGroups=11, usedMetadataFile=true,
cacheFileRoot=/drill/testdata/metadata_cache/store_sales_null_blocks_int,
columns=[`int_nulls_id`, `int_id`, `ss_ticket_number`, `extra`]]])
{noformat}
This is a regression from Drill 1.15.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-7225) Merging of columnTypeInfo for file with different schema throws NullPointerException during refresh metadata
Venkata Jyothsna Donapati created DRILL-7225: Summary: Merging of columnTypeInfo for file with different schema throws NullPointerException during refresh metadata Key: DRILL-7225 URL: https://issues.apache.org/jira/browse/DRILL-7225 Project: Apache Drill Issue Type: Bug Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7199) Optimize the time taken to populate column statistics for non-interesting columns
Venkata Jyothsna Donapati created DRILL-7199: Summary: Optimize the time taken to populate column statistics for non-interesting columns Key: DRILL-7199 URL: https://issues.apache.org/jira/browse/DRILL-7199 Project: Apache Drill Issue Type: Bug Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Currently populating column statistics for non-existent columns very long since it is populated for every row group. Since non-existent column statistics are common for the table, it can be populated once and can be reused. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7171) Count(*) query on leaf level directory is not reading summary cache file.
Venkata Jyothsna Donapati created DRILL-7171: Summary: Count(*) query on leaf level directory is not reading summary cache file. Key: DRILL-7171 URL: https://issues.apache.org/jira/browse/DRILL-7171 Project: Apache Drill Issue Type: Bug Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Since the leaf level directory doesn't store the metadata directories file, while reading summary if the directories cache file is not present, it is assumed that the cache is possibly corrupt and reading of the summary cache file is skipped. Metadata directories cache file should be created at the leaf level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7066) Auto-refresh should pick up existing columns from metadata cache
Venkata Jyothsna Donapati created DRILL-7066: Summary: Auto-refresh should pick up existing columns from metadata cache Key: DRILL-7066 URL: https://issues.apache.org/jira/browse/DRILL-7066 Project: Apache Drill Issue Type: Sub-task Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Fix For: 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7065) Ensure backward compatibility is maintained
Venkata Jyothsna Donapati created DRILL-7065: Summary: Ensure backward compatibility is maintained Key: DRILL-7065 URL: https://issues.apache.org/jira/browse/DRILL-7065 Project: Apache Drill Issue Type: Sub-task Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Fix For: 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7063) Create separate summary file for schema, totalRowCount, totalNullCount (includes maintenance)
Venkata Jyothsna Donapati created DRILL-7063: Summary: Create separate summary file for schema, totalRowCount, totalNullCount (includes maintenance) Key: DRILL-7063 URL: https://issues.apache.org/jira/browse/DRILL-7063 Project: Apache Drill Issue Type: Sub-task Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Fix For: 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7064) Leverage the summary's totalRowCount and totalNullCount for COUNT() queries (also prevent eager expansion of files)
Venkata Jyothsna Donapati created DRILL-7064: Summary: Leverage the summary's totalRowCount and totalNullCount for COUNT() queries (also prevent eager expansion of files) Key: DRILL-7064 URL: https://issues.apache.org/jira/browse/DRILL-7064 Project: Apache Drill Issue Type: Sub-task Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Fix For: 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7062) Run-time row group pruning
Venkata Jyothsna Donapati created DRILL-7062: Summary: Run-time row group pruning Key: DRILL-7062 URL: https://issues.apache.org/jira/browse/DRILL-7062 Project: Apache Drill Issue Type: Sub-task Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Boaz Ben-Zvi Fix For: 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7058) Refresh command to support subset of columns
Venkata Jyothsna Donapati created DRILL-7058: Summary: Refresh command to support subset of columns Key: DRILL-7058 URL: https://issues.apache.org/jira/browse/DRILL-7058 Project: Apache Drill Issue Type: New Feature Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Fix For: 1.16.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-7028) Reduce the planning time of queries on large Parquet tables with large metadata cache files
Venkata Jyothsna Donapati created DRILL-7028: Summary: Reduce the planning time of queries on large Parquet tables with large metadata cache files Key: DRILL-7028 URL: https://issues.apache.org/jira/browse/DRILL-7028 Project: Apache Drill Issue Type: Improvement Components: Metadata Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati Fix For: 1.16.0 If the Parquet table has a large number of small files, the metadata cache files grow larger and the planner tries to read the large metadata cache file which leads to the planning time overhead. Most of the time of execution is spent during the planning phase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6822) Fix imports for DRILL-6381
Venkata Jyothsna Donapati created DRILL-6822: Summary: Fix imports for DRILL-6381 Key: DRILL-6822 URL: https://issues.apache.org/jira/browse/DRILL-6822 Project: Apache Drill Issue Type: Task Reporter: Venkata Jyothsna Donapati Assignee: Venkata Jyothsna Donapati -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6261) logging "Waiting for X queries to complete before shutting down" even before shutdown request is triggered
Venkata Jyothsna Donapati created DRILL-6261: Summary: logging "Waiting for X queries to complete before shutting down" even before shutdown request is triggered Key: DRILL-6261 URL: https://issues.apache.org/jira/browse/DRILL-6261 Project: Apache Drill Issue Type: Bug Reporter: Venkata Jyothsna Donapati After https://issues.apache.org/jira/browse/DRILL-5922 changes "Waiting for X queries to complete before shutting down" is logged every time a query runs instead of it being logged after a shutdown request is triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6252) Foreman node is going down when the non foreman node is stopped
Venkata Jyothsna Donapati created DRILL-6252: Summary: Foreman node is going down when the non foreman node is stopped Key: DRILL-6252 URL: https://issues.apache.org/jira/browse/DRILL-6252 Project: Apache Drill Issue Type: Bug Reporter: Venkata Jyothsna Donapati Two drillbits are running. I'm running a join query over parquet and tried to stop the non-foreman node using drillbit.sh stop. The query fails with *"Error: DATA_READ ERROR: Exception occurred while reading from disk".* The non-foreman node goes down. The foreman node also goes down. When I looked at the drillbit.log of both foreman and non-foreman I found that there is memory leak "Memory was leaked by query. Memory leaked: (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are the stack traces for memory leaks [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (3145728) Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 (res/actual/peak/limit) Fragment 2:1 [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (3145728) Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 (res/actual/peak/limit) Ping me for the logs and more information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6245) Clicking on anything redirects to main login page
Venkata Jyothsna Donapati created DRILL-6245: Summary: Clicking on anything redirects to main login page Key: DRILL-6245 URL: https://issues.apache.org/jira/browse/DRILL-6245 Project: Apache Drill Issue Type: Improvement Reporter: Venkata Jyothsna Donapati When the Drill Web UI is accessed using https and then by http protocol, the Web UI is always trying to redirect to main login page if anything is clicked on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6244) Ability to shutdown cross domain Drillbits
Venkata Jyothsna Donapati created DRILL-6244: Summary: Ability to shutdown cross domain Drillbits Key: DRILL-6244 URL: https://issues.apache.org/jira/browse/DRILL-6244 Project: Apache Drill Issue Type: Improvement Reporter: Venkata Jyothsna Donapati Currently Drill Web UI supports shutdown of current Drillbit and doesn't provide an option to shutdown cross domain Drillbits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6243) Alert box to confirm shutdown of drillbit
Venkata Jyothsna Donapati created DRILL-6243: Summary: Alert box to confirm shutdown of drillbit Key: DRILL-6243 URL: https://issues.apache.org/jira/browse/DRILL-6243 Project: Apache Drill Issue Type: Improvement Reporter: Venkata Jyothsna Donapati -- This message was sent by Atlassian JIRA (v7.6.3#76005)
