[jira] [Resolved] (IMPALA-6475) Enable running TPCH on Kudu in our nightly tests

2018-04-27 Thread Michael Brown (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Brown resolved IMPALA-6475.
---
   Resolution: Fixed
Fix Version/s: Impala 2.12.0

> Enable running TPCH on Kudu in our nightly tests
> 
>
> Key: IMPALA-6475
> URL: https://issues.apache.org/jira/browse/IMPALA-6475
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.11.0
>Reporter: Taras Bobrovytsky
>Priority: Major
> Fix For: Impala 2.12.0
>
>
> It looks like we do not run TPCH and TPCDS on Kudu at all. With TPCDS, this 
> is intentional because Kudu does not support Decimal. However, we should run 
> TPCH on Kudu.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6885) test_recover_partitions.py test_support_all_types fails

2018-04-27 Thread Michael Brown (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Brown resolved IMPALA-6885.
---
   Resolution: Fixed
Fix Version/s: Impala 2.13.0

> test_recover_partitions.py test_support_all_types fails
> ---
>
> Key: IMPALA-6885
> URL: https://issues.apache.org/jira/browse/IMPALA-6885
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Vuk Ercegovac
>Assignee: Vuk Ercegovac
>Priority: Blocker
> Fix For: Impala 2.13.0
>
>
> {noformat}
> metadata/test_recover_partitions.py:332: in test_support_all_types
> assert len(result.data) == (old_length + 1),\
> E   AssertionError: ALTER TABLE 
> test_support_all_types_bc3b11e9.test_recover_partitions RECOVER PARTITIONS 
> failed to handle some data types.
> E   assert 2 == (2 + 1)
> E+  where 2 = len(['1\t2\t3\t4\t55.55\t6.59904632568\t7.7\tj
> \tk\ts\t-1\t1\t2B\tNOT CACHED\tNOT 
> CACHED\tTEXT\tfalse\ts3a://impala...s/a=1/b=2/c=3/d=4/e=55.55/f=6.59904632568/g=7.7/j=j
> /k=k/s=s', 'Total\t\t\t\t\t\t\t\t\t\t-1\t1\t2B\t0B\t\t\t\t'])
> E+where ['1\t2\t3\t4\t55.55\t6.59904632568\t7.7\tj
> \tk\ts\t-1\t1\t2B\tNOT CACHED\tNOT 
> CACHED\tTEXT\tfalse\ts3a://impala...s/a=1/b=2/c=3/d=4/e=55.55/f=6.59904632568/g=7.7/j=j
> /k=k/s=s', 'Total\t\t\t\t\t\t\t\t\t\t-1\t1\t2B\t0B\t\t\t\t'] = 
>  0x722d150>.data{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6947) kudu: GetTableLocations RPC timing out with ASAN

2018-04-27 Thread Michael Brown (JIRA) 
Michael Brown created IMPALA-6947:
-

 Summary: kudu: GetTableLocations RPC timing out with ASAN
 Key: IMPALA-6947
 URL: https://issues.apache.org/jira/browse/IMPALA-6947
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Michael Brown
Assignee: Thomas Tauber-Marshall


{noformat}
query_test/test_kudu.py:84: in test_kudu_insert
self.run_test_case('QueryTest/kudu_insert', vector, use_db=unique_database)
common/impala_test_suite.py:398: in run_test_case
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:613: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:160: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:173: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:341: in __execute_query
self.wait_for_completion(handle)
beeswax/impala_beeswax.py:361: in wait_for_completion
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E   ImpalaBeeswaxException: ImpalaBeeswaxException:
EQuery aborted:Kudu error(s) reported, first error: Timed out: 
GetTableLocations { table: 'impala::test_kudu_insert_70eff904.kudu_test', 
partition-key: (HASH (a, b): 2), attempt: 1 } failed: GetTableLocations RPC to 
127.0.0.1:7051 timed out after 10.000s (SENT)
E   
E   Key already present in Kudu table 
'impala::test_kudu_insert_70eff904.kudu_test'. (1 of 3 similar)
E   Error in Kudu table 'impala::test_kudu_insert_70eff904.kudu_test': Timed 
out: GetTableLocations { table: 'impala::test_kudu_insert_70eff904.kudu_test', 
partition-key: (HASH (a, b): 2), attempt: 1 } failed: GetTableLocations RPC to 
127.0.0.1:7051 timed out after 10.000s (SENT) (1 of 21 similar)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6902) query_test.test_udfs.TestUdfExecution.test_native_functions_race failed during core/thrift build

2018-04-27 Thread Vuk Ercegovac (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vuk Ercegovac resolved IMPALA-6902.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> query_test.test_udfs.TestUdfExecution.test_native_functions_race failed 
> during core/thrift build
> 
>
> Key: IMPALA-6902
> URL: https://issues.apache.org/jira/browse/IMPALA-6902
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: David Knupp
>Assignee: Vuk Ercegovac
>Priority: Blocker
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Assigning to Vuk, who authored this test as part of the patch for IMPALA-6488:
> https://gerrit.cloudera.org/c/9626/
> I'm not sure that this really the same failure though, so I'm not reopening 
> that earlier JIRA. If I'm mistaken, please feel free to reopen/reassign as 
> necessary.
> Stacktrace
> {noformat}
> query_test/test_udfs.py:377: in test_native_functions_race
>  assert len(errors) == 0
> E assert 1 == 0
> E + where 1 = len([ImpalaBeeswaxException()])
> Standard Output
> ImpalaBeeswaxException:
>  INNER EXCEPTION: 
>  MESSAGE: ImpalaRuntimeException: Error making 'alterDatabase' RPC to Hive 
> Metastore: 
> CAUSED BY: NoSuchObjectException: test_native_functions_race_fc9680e5: 
> Transaction rolled back due to failure during commit
> {noformat}
> Standard Error
> {noformat}
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_native_functions_race_fc9680e5` CASCADE;
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_native_functions_race_fc9680e5`;
> MainThread: Created database "test_native_functions_race_fc9680e5" for test 
> ID 
> "query_test/test_udfs.py::TestUdfExecution::()::test_native_functions_race[exec_option:
>  {'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 
> 'exec_single_node_rows_threshold': 0, 'enable_expr_rewrites': True} | 
> table_format: text/none]"
> -- connecting to: localhost:21000
> -- executing against localhost:21000
> create function test_native_functions_race_fc9680e5.use_it(string) returns 
> string
>  LOCATION '/test-warehouse/libTestUdfs.so'
>  SYMBOL='_Z8IdentityPN10impala_udf15FunctionContextERKNS_9StringValE';
> {noformat}
> From catalogd log: 
> {noformat}
> I0420 14:54:00.014191 19585 jni-util.cc:230] 
> org.apache.impala.common.ImpalaRuntimeException: Error making 'alterDatabase' 
> RPC to Hive Metastore:
> at 
> org.apache.impala.service.CatalogOpExecutor.applyAlterDatabase(CatalogOpExecutor.java:2770)
> at 
> org.apache.impala.service.CatalogOpExecutor.dropFunction(CatalogOpExecutor.java:1521)
> at 
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:307)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: NoSuchObjectException(message:test_native_functions_race_fc9680e5: 
> Transaction rolled back due to failure during commit)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_database_result$alter_database_resultStandardScheme.read(ThriftHiveMetastore.java:20111)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_database_result$alter_database_resultStandardScheme.read(ThriftHiveMetastore.java:20088)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_database_result.read(ThriftHiveMetastore.java:20030)
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_database(ThriftHiveMetastore.java:814)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_database(ThriftHiveMetastore.java:800)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alterDatabase(HiveMetaStoreClient.java:1420)
> at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
> at com.sun.proxy.$Proxy5.alterDatabase(Unknown Source)
> at 
> org.apache.impala.service.CatalogOpExecutor.applyAlterDatabase(CatalogOpExecutor.java:2768)
> ... 3 more
> I0420 14:54:01.581200  3230 catalog-server.cc:245] A catalog update with 5 
> entries is assembled. Catalog version: 14385 Last sent catalog version: 14377
> I0420 14:54:01.582358  3225 catalog-server.cc:480] Collected deletion: 
>

[jira] [Closed] (IMPALA-6407) [DOCS] Clean up subheadings under Fixed Issues

2018-04-27 Thread Alex Rodoni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rodoni closed IMPALA-6407.
---
Resolution: Won't Fix

> [DOCS] Clean up subheadings under Fixed Issues
> --
>
> Key: IMPALA-6407
> URL: https://issues.apache.org/jira/browse/IMPALA-6407
> Project: IMPALA
>  Issue Type: Improvement
>Reporter: John Russell
>Assignee: John Russell
>Priority: Major
>
> The fixed issues page: 
> http://impala.apache.org/docs/build/html/topics/impala_fixed_issues.html
> has a number of blank subheadings, because of downstream CDH maintenance 
> releases that didn't have an actual upstream counterpart. Remove these empty 
> subheadings.
> Also the 2.11 fixed issues subheading has the text, "Issues Fixed in Impala 
> 2.1.10". Looks like the substitution variable conflicted because of ambiguity 
> around the suffix '2110'. Straighten that out by referencing the correct 
> impala2_11_0 variable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6946) Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue

2018-04-27 Thread Tim Armstrong (JIRA) 
Tim Armstrong created IMPALA-6946:
-

 Summary: Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
 Key: IMPALA-6946
 URL: https://issues.apache.org/jira/browse/IMPALA-6946
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 2.13.0, 
Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


The bug comes from conversion between signed and unsigned.

{noformat}
 DCHECK_GT(num_repeats_to_consume, 0);

(gdb) p num_repeats_to_consume
$2 = -1003251240
{noformat}

{noformat}
#2  0x01f422f0 in impala::ScalarColumnReader::ReadNonRepeatedValueBatch (this=0x6f547d50, 
pool=0x7fb37bdbb330, max_values=0, 
tuple_size=90, tuple_mem=0x0, num_values=0x7fb37bdbb330) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
#3  0x03fceea7 in google::LogMessage::Flush() ()
#4  0x03fd246e in google::LogMessageFatal::~LogMessageFatal() ()
#5  0x01f97197 in impala::RleBatchDecoder::GetRepeatedValue (this=0x1803fc6a0, num_repeats_to_consume=-1003251240)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/rle-encoding.h:493
#6  0x01f96d7b in 
impala::DictDecoder::DecodeNextValue (this=0x1803fc698, 
value=0x16b7e9000)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:455
#7  0x01f94528 in 
impala::DictDecoder::GetNextValue (value=0x16b7e9000, 
this=0x1803fc698)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:446
#8  impala::ScalarColumnReader::ReadSlot (pool=0x6f547d98, tuple=0x16b7e9000, 
this=0x1803fc400)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:499
#9  impala::ScalarColumnReader::MaterializeValueBatch (this=0x1803fc400, 
pool=0x6f547d98, max_values=1, tuple_size=90, 
tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:413
#10 0x01f67b4a in impala::ScalarColumnReader::MaterializeValueBatch 
(this=0x1803fc400, pool=0x6f547d98, max_values=1, 
tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:436
#11 0x01f657c2 in impala::ScalarColumnReader::ReadValueBatch (this=0x1803fc400, 
pool=0x6f547d98, max_values=1, tuple_size=90, 
tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:354
#12 0x01f422f0 in impala::ScalarColumnReader::ReadNonRepeatedValueBatch (this=0x1803fc400, 
pool=0x6f547d98, max_values=1, 
tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
#13 0x01d70a1c in impala::HdfsParquetScanner::AssembleRows 
(this=0x1a926000, column_readers=..., row_batch=0xdfc0780, 
skip_row_group=0x1a9261b8)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:936
#14 0x01d6d4f7 in impala::HdfsParquetScanner::GetNextInternal 
(this=0x1a926000, row_batch=0xdfc0780)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:434
#15 0x01d6b6c6 in impala::HdfsParquetScanner::ProcessSplit 
(this=0x1a926000) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:332
#16 0x01cf30e4 in impala::HdfsScanNode::ProcessSplit (this=0x10c21000, 
filter_ctxs=..., expr_results_pool=0x7fb37bdbc480, scan_range=0xf3276c0, 
scanner_thread_reservation=0x7fb37bdbc400)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:482
#17 0x01cf2476 in impala::HdfsScanNode::ScannerThread (this=0x10c21000, 
scanner_thread_reservation=90112)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:385
#18 0x01cf18a8 in impala::HdfsScanNode::::operator()(void) 
const (__closure=0x7fb37bdbcbc8)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:300

[jira] [Resolved] (IMPALA-6945) IMPALA-6892 doesn't clean pick to 2.x

2018-04-27 Thread Tim Armstrong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6945.
---
   Resolution: Fixed
Fix Version/s: Impala 2.13.0

> IMPALA-6892 doesn't clean pick to 2.x
> -
>
> Key: IMPALA-6945
> URL: https://issues.apache.org/jira/browse/IMPALA-6945
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Michael Brown
>Assignee: Tim Armstrong
>Priority: Blocker
> Fix For: Impala 2.13.0
>
>
> {noformat}
> mikeb@mikeb-ub16:1:~/Impala (2.x|CHERRY-PICKING) $ git diff
> diff --cc be/src/exec/exec-node.cc
> index c94f38c,d2dd85f..000
> --- a/be/src/exec/exec-node.cc
> +++ b/be/src/exec/exec-node.cc
> @@@ -68,11 -68,6 +68,14 @@@
>   using strings::Substitute;
> ++<<< HEAD
>  +DECLARE_int32(be_port);
>  +DECLARE_string(hostname);
>  +DEFINE_bool_hidden(enable_partitioned_hash_join, true, "Deprecated - has no 
> effect");
>  +DEFINE_bool_hidden(enable_partitioned_aggregation, true, "Deprecated - has 
> no effect");
>  +
> ++===
> ++>>> 518bcd3... IMPALA-6892: CheckHashAndDecrypt() includes file and host
>   namespace impala {
>   const string ExecNode::ROW_THROUGHPUT_COUNTER = "RowsReturnedRate";
> mikeb@mikeb-ub16:0:~/Impala (2.x|CHERRY-PICKING) $
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6821) Push down limits into Kudu

2018-04-27 Thread Thomas Tauber-Marshall (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Tauber-Marshall resolved IMPALA-6821.

Resolution: Fixed

> Push down limits into Kudu
> --
>
> Key: IMPALA-6821
> URL: https://issues.apache.org/jira/browse/IMPALA-6821
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Reporter: Andrew Wong
>Assignee: Thomas Tauber-Marshall
>Priority: Major
>  Labels: kudu, perf
>
> The progress made in KUDU-16 introduced a way to impose limits on Kudu 
> scanners, potentially reducing the number of RPCs sent per scan, CPU used for 
> scanner evaluation on the tablet server, etc. It would be nice if Impala 
> could make use of this new behavior.
> I put up an admittedly clunky [patch|https://gerrit.cloudera.org/c/9923/] 
> that implements limits on a per-token basis. I'm no Impala expert, so there 
> may be an API missing in Kudu that would make Impala's life easier in 
> implementing this. Maybe it's as easy as adjusting the limit after 
> re-hydrating the token into a scanner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6949) Make it possible to start the minicluster with HDFS erasure coding enabled

2018-04-27 Thread Taras Bobrovytsky (JIRA) 
Taras Bobrovytsky created IMPALA-6949:
-

 Summary: Make it possible to start the minicluster with HDFS 
erasure coding enabled
 Key: IMPALA-6949
 URL: https://issues.apache.org/jira/browse/IMPALA-6949
 Project: IMPALA
  Issue Type: Task
  Components: Infrastructure
Affects Versions: Impala 3.0
Reporter: Taras Bobrovytsky
Assignee: Taras Bobrovytsky


We need to add the option to start the mini cluster with 5 nodes and erasure 
coding enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (IMPALA-6946) Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue

2018-04-27 Thread Tim Armstrong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong updated IMPALA-6946:
--
Target Version: Impala 2.13.0, Impala 3.1.0

> Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
> -
>
> Key: IMPALA-6946
> URL: https://issues.apache.org/jira/browse/IMPALA-6946
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 2.13.0, 
> Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: crash
>
> The bug comes from conversion between signed and unsigned.
> {noformat}
>  DCHECK_GT(num_repeats_to_consume, 0);
> (gdb) p num_repeats_to_consume
> $2 = -1003251240
> {noformat}
> {noformat}
> #2  0x01f422f0 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x6f547d50, 
> pool=0x7fb37bdbb330, max_values=0, 
> tuple_size=90, tuple_mem=0x0, num_values=0x7fb37bdbb330) at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
> #3  0x03fceea7 in google::LogMessage::Flush() ()
> #4  0x03fd246e in google::LogMessageFatal::~LogMessageFatal() ()
> #5  0x01f97197 in impala::RleBatchDecoder int>::GetRepeatedValue (this=0x1803fc6a0, num_repeats_to_consume=-1003251240)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/rle-encoding.h:493
> #6  0x01f96d7b in 
> impala::DictDecoder::DecodeNextValue (this=0x1803fc698, 
> value=0x16b7e9000)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:455
> #7  0x01f94528 in 
> impala::DictDecoder::GetNextValue (value=0x16b7e9000, 
> this=0x1803fc698)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:446
> #8  impala::ScalarColumnReader true>::ReadSlot (pool=0x6f547d98, tuple=0x16b7e9000, 
> this=0x1803fc400)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:499
> #9  impala::ScalarColumnReader true>::MaterializeValueBatch (this=0x1803fc400, 
> pool=0x6f547d98, max_values=1, tuple_size=90, 
> tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:413
> #10 0x01f67b4a in impala::ScalarColumnReader (parquet::Type::type)6, true>::MaterializeValueBatch 
> (this=0x1803fc400, pool=0x6f547d98, max_values=1, 
> tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:436
> #11 0x01f657c2 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadValueBatch (this=0x1803fc400, 
> pool=0x6f547d98, max_values=1, tuple_size=90, 
> tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:354
> #12 0x01f422f0 in impala::ScalarColumnReader (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x1803fc400, 
> pool=0x6f547d98, max_values=1, 
> tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
> #13 0x01d70a1c in impala::HdfsParquetScanner::AssembleRows 
> (this=0x1a926000, column_readers=..., row_batch=0xdfc0780, 
> skip_row_group=0x1a9261b8)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:936
> #14 0x01d6d4f7 in impala::HdfsParquetScanner::GetNextInternal 
> (this=0x1a926000, row_batch=0xdfc0780)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:434
> #15 0x01d6b6c6 in impala::HdfsParquetScanner::ProcessSplit 
> (this=0x1a926000) at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:332
> #16 0x01cf30e4 in impala::HdfsScanNode::ProcessSplit 
> (this=0x10c21000, filter_ctxs=..., expr_results_pool=0x7fb37bdbc480, 
> scan_range=0xf3276c0, scanner_thread_reservation=0x7fb37bdbc400)
> at 
> /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:482
> #17

[jira] [Created] (IMPALA-6948) Coordinators don't detect the deletion of tables that occurred outside of impala after catalog restart

2018-04-27 Thread Dimitris Tsirogiannis (JIRA) 
Dimitris Tsirogiannis created IMPALA-6948:
-

 Summary: Coordinators don't detect the deletion of tables that 
occurred outside of impala after catalog restart
 Key: IMPALA-6948
 URL: https://issues.apache.org/jira/browse/IMPALA-6948
 Project: IMPALA
  Issue Type: Bug
  Components: Catalog
Affects Versions: Impala 3.0, Impala 2.12.0
Reporter: Dimitris Tsirogiannis
Assignee: Dimitris Tsirogiannis


Upon catalog restart the coordinators detect this event and request a full 
topic update from the statestore. In certain cases, the topic update protocol 
executed between the statestore and the catalog fails to detect catalog objects 
that were deleted from the Metastore externally (e.g. via HIVE), thus causing 
these objects to show up again in each coordinator's catalog cache. The end 
result is that the catalog server and the coordinator's cache are out of sync 
and in some cases the only solution is to restart both the catalog and the 
statestore. 

The following sequence can reproduce this issue:
{code:java}
impala> create table lala(int a);
bash> kill -9 `pidof catalogd`
hive> drop table lala;
bash> restart catalogd 
impala> show tables;
--- lala shows up in the list of tables;{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6314) Add run time scalar subquery check for uncorrelated subqueries

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457046#comment-16457046
 ] 

ASF subversion and git services commented on IMPALA-6314:
-

Commit 1e79f14798a8f742bbf17f79a427dcef3faf in impala's branch 
refs/heads/master from [~boroknagyz]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1e79f14 ]

IMPALA-6314: Add run time scalar subquery check for uncorrelated subqueries

If a scalar subquery is used with a binary predicate,
or, used in an arithmetic expression, it must return
only one row/column to be valid. If this cannot be
guaranteed at parse time through a single row aggregate
or limit clause, Impala fails the query like such.

E.g., currently the following query is not allowed:
SELECT bigint_col
FROM alltypesagg
WHERE id = (SELECT id FROM alltypesagg WHERE id = 1)

However, it would be allowed if the query contained
a LIMIT 1 clause, or instead of id it was max(id).

This commit makes the example valid by introducing a
runtime check to test if the subquery returns a single
row. If the subquery returns more than one row, it
aborts the query with an error.

I added a new node type, called CardinalityCheckNode. It
is created during planning on top of the subquery when
needed, then during execution it checks if its child
only returns a single row.

I extended the frontend tests and e2e tests as well.

Change-Id: I0f52b93a60eeacedd242a2f17fa6b99c4fc38e06
Reviewed-on: http://gerrit.cloudera.org:8080/9005
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins 


> Add run time scalar subquery check for uncorrelated subqueries
> --
>
> Key: IMPALA-6314
> URL: https://issues.apache.org/jira/browse/IMPALA-6314
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Backend, Frontend
>Reporter: Tim Armstrong
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>  Labels: planner, ramp-up, tpc-ds
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6532) NullPointerException in HiveContextAwareRecordReader.initIOContext() when executing Hive query

2018-04-27 Thread Michael Brown (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457100#comment-16457100
 ] 

Michael Brown commented on IMPALA-6532:
---

This was while loading TPCH during data load. The failure was in  {{INSERT 
OVERWRITE TABLE tpch_text_gzip.lineitem SELECT * FROM tpch.lineitem}} and the 
exception was
{noformat}
java.lang.Exception: java.lang.NullPointerException
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.initIOContext(HiveContextAwareRecordReader.java:171)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.initIOContext(HiveContextAwareRecordReader.java:208)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:258)
at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:438)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

> NullPointerException in HiveContextAwareRecordReader.initIOContext() when 
> executing Hive query
> --
>
> Key: IMPALA-6532
> URL: https://issues.apache.org/jira/browse/IMPALA-6532
> Project: IMPALA
>  Issue Type: Bug
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Bikramjeet Vig
>Assignee: Joe McDonnell
>Priority: Blocker
>  Labels: broken-build, flaky
>
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1273/console]
> {noformat}
> 02:48:03 [gw15] FAILED 
> metadata/test_partition_metadata.py::TestPartitionMetadata::test_partition_metadata_compatibility[exec_option:
>  {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
> {noformat}
> {noformat}
> 03:29:11 === FAILURES 
> ===
> 03:29:11  
> TestPartitionMetadata.test_partition_metadata_compatibility[exec_option: 
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
> 03:29:11 [gw15] linux2 -- Python 2.7.12 
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 03:29:11 metadata/test_partition_metadata.py:127: in 
> test_partition_metadata_compatibility
> 03:29:11 "insert into %s partition (x) values(1,1)" % FQ_TBL_HIVE)
> 03:29:11 common/impala_test_suite.py:684: in run_stmt_in_hive
> 03:29:11 raise RuntimeError(stderr)
> 03:29:11 E   RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
> 03:29:11 E   SLF4J: Found binding in 
> [jar:file:/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.15.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 03:29:11 E   SLF4J: Found binding in 
> [jar:file:/home/ubuntu/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.15.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 03:29:11 E   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for 
> an explanation.
> 03:29:11 E   SLF4J: Actual binding is of type 
> [org.slf4j.impl.Log4jLoggerFactory]
> 03:29:11 E   scan complete in 3ms
> 03:29:11 E   Connecting to jdbc:hive2://localhost:11050
> 03:29:11 E   Connected to: Apache Hive (version 1.1.0-cdh5.15.0-SNAPSHOT)
> 03:29:11 E   Driver: Hive JDBC (version 1.1.0-cdh5.15.0-SNAPSHOT)
> 03:29:11 E   Transaction isolation: TRANSACTION_REPEATABLE_READ
> 03:29:11 E   No rows affected (0.045 seconds)
> 03:29:11 E   INFO  : Compiling 
> command(queryId=ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba): 
> insert into 
> test_partition_metadata_compatibility_b2ac5e.part_parquet_tbl_hive partition 
> (x) values(1,1)
> 03:29:11 E   INFO  : Semantic Analysis Completed
> 03:29:11 E   INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), 
> FieldSchema(name:_col1,

[jira] [Commented] (IMPALA-6821) Push down limits into Kudu

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457165#comment-16457165
 ] 

ASF subversion and git services commented on IMPALA-6821:
-

Commit 87be63e321f688486b98d4ea69200967a8a2effa in impala's branch 
refs/heads/master from [~twmarshall]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=87be63e ]

IMPALA-6821: Push down limits into Kudu

This patch takes advantage of a recent change in Kudu (KUDU-16) that
exposes the ability to set limits on KuduScanners. Since each
KuduScanner corresponds to a scan token, and there will be multiple
scan tokens per query, this is just a performance optimization in
cases where the limit is smaller than the number of rows per token,
and Impala still needs to apply the limit on our side for cases where
the limit is greater than the number of rows per token.

Testing:
- Added e2e tests for various situations where limits are applied at
  a Kudu scan node.
- For the query 'select * from tpch_kudu.lineitem limit 1', a best
  case perf scenario for this change where the limit is highly
  effective, the time spent in the Kudu scan node was reduced from
  6.107ms to 3.498ms (avg over 3 runs).
- For the query 'select count(*) from (select * from
  tpch_kudu.lineitem limit 100) v', a worst case perf scenario for
  this change where the limit is ineffective, the time spent in the
  Kudu scan node was essentially unchanged, 32.815ms previously vs.
  29.532ms (avg over 3 runs).

Change-Id: Ibe35e70065d8706b575e24fe20902cd405b49941
Reviewed-on: http://gerrit.cloudera.org:8080/10119
Reviewed-by: Thomas Tauber-Marshall 
Tested-by: Impala Public Jenkins 


> Push down limits into Kudu
> --
>
> Key: IMPALA-6821
> URL: https://issues.apache.org/jira/browse/IMPALA-6821
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Reporter: Andrew Wong
>Assignee: Thomas Tauber-Marshall
>Priority: Major
>  Labels: kudu, perf
>
> The progress made in KUDU-16 introduced a way to impose limits on Kudu 
> scanners, potentially reducing the number of RPCs sent per scan, CPU used for 
> scanner evaluation on the tablet server, etc. It would be nice if Impala 
> could make use of this new behavior.
> I put up an admittedly clunky [patch|https://gerrit.cloudera.org/c/9923/] 
> that implements limits on a per-token basis. I'm no Impala expert, so there 
> may be an API missing in Kudu that would make Impala's life easier in 
> implementing this. Maybe it's as easy as adjusting the limit after 
> re-hydrating the token into a scanner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Resolved] (IMPALA-6821) Push down limits into Kudu

2018-04-27 Thread Thomas Tauber-Marshall (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Tauber-Marshall resolved IMPALA-6821.

Resolution: Fixed

> Push down limits into Kudu
> --
>
> Key: IMPALA-6821
> URL: https://issues.apache.org/jira/browse/IMPALA-6821
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Reporter: Andrew Wong
>Assignee: Thomas Tauber-Marshall
>Priority: Major
>  Labels: kudu, perf
>
> The progress made in KUDU-16 introduced a way to impose limits on Kudu 
> scanners, potentially reducing the number of RPCs sent per scan, CPU used for 
> scanner evaluation on the tablet server, etc. It would be nice if Impala 
> could make use of this new behavior.
> I put up an admittedly clunky [patch|https://gerrit.cloudera.org/c/9923/] 
> that implements limits on a per-token basis. I'm no Impala expert, so there 
> may be an API missing in Kudu that would make Impala's life easier in 
> implementing this. Maybe it's as easy as adjusting the limit after 
> re-hydrating the token into a scanner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Updated] (IMPALA-6532) NullPointerException in HiveContextAwareRecordReader.initIOContext() when executing Hive query

2018-04-27 Thread Michael Brown (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Brown updated IMPALA-6532:
--
Priority: Blocker  (was: Critical)

> NullPointerException in HiveContextAwareRecordReader.initIOContext() when 
> executing Hive query
> --
>
> Key: IMPALA-6532
> URL: https://issues.apache.org/jira/browse/IMPALA-6532
> Project: IMPALA
>  Issue Type: Bug
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Bikramjeet Vig
>Assignee: Joe McDonnell
>Priority: Blocker
>  Labels: broken-build, flaky
>
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1273/console]
> {noformat}
> 02:48:03 [gw15] FAILED 
> metadata/test_partition_metadata.py::TestPartitionMetadata::test_partition_metadata_compatibility[exec_option:
>  {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
> {noformat}
> {noformat}
> 03:29:11 === FAILURES 
> ===
> 03:29:11  
> TestPartitionMetadata.test_partition_metadata_compatibility[exec_option: 
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
> 03:29:11 [gw15] linux2 -- Python 2.7.12 
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 03:29:11 metadata/test_partition_metadata.py:127: in 
> test_partition_metadata_compatibility
> 03:29:11 "insert into %s partition (x) values(1,1)" % FQ_TBL_HIVE)
> 03:29:11 common/impala_test_suite.py:684: in run_stmt_in_hive
> 03:29:11 raise RuntimeError(stderr)
> 03:29:11 E   RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
> 03:29:11 E   SLF4J: Found binding in 
> [jar:file:/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.15.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 03:29:11 E   SLF4J: Found binding in 
> [jar:file:/home/ubuntu/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.15.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 03:29:11 E   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for 
> an explanation.
> 03:29:11 E   SLF4J: Actual binding is of type 
> [org.slf4j.impl.Log4jLoggerFactory]
> 03:29:11 E   scan complete in 3ms
> 03:29:11 E   Connecting to jdbc:hive2://localhost:11050
> 03:29:11 E   Connected to: Apache Hive (version 1.1.0-cdh5.15.0-SNAPSHOT)
> 03:29:11 E   Driver: Hive JDBC (version 1.1.0-cdh5.15.0-SNAPSHOT)
> 03:29:11 E   Transaction isolation: TRANSACTION_REPEATABLE_READ
> 03:29:11 E   No rows affected (0.045 seconds)
> 03:29:11 E   INFO  : Compiling 
> command(queryId=ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba): 
> insert into 
> test_partition_metadata_compatibility_b2ac5e.part_parquet_tbl_hive partition 
> (x) values(1,1)
> 03:29:11 E   INFO  : Semantic Analysis Completed
> 03:29:11 E   INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), 
> FieldSchema(name:_col1, type:string, comment:null)], properties:null)
> 03:29:11 E   INFO  : Completed compiling 
> command(queryId=ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba); 
> Time taken: 0.123 seconds
> 03:29:11 E   INFO  : Executing 
> command(queryId=ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba): 
> insert into 
> test_partition_metadata_compatibility_b2ac5e.part_parquet_tbl_hive partition 
> (x) values(1,1)
> 03:29:11 E   INFO  : Query ID = 
> ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba
> 03:29:11 E   INFO  : Total jobs = 3
> 03:29:11 E   INFO  : Launching Job 1 out of 3
> 03:29:11 E   INFO  : Starting task [Stage-1:MAPRED] in serial mode
> 03:29:11 E   INFO  : Number of reduce tasks is set to 0 since there's no 
> reduce operator
> 03:29:11 E   INFO  : number of splits:1
> 03:29:11 E   INFO  : Submitting tokens for job: job_local1716894275_0002
> 03:29:11 E   INFO  : The url to track the job: http://localhost:8080/
> 03:29:11 E   INFO  : Job running in-process (local Hadoop)
> 03:29:11 E   INFO  : 2018-02-15 02:48:03,500 Stage-1 map = 0%,  reduce = 0%
> 03:29:11 E   ERROR : Ended Job = job_local1716894275_0002 with errors
> 03:29:11 E   ERROR : FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> 03:29:11 E   INFO  : MapReduce Jobs Launched: 
> 03:29:11 E   INFO  : Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 FAIL
> 03:29:11 E   INFO  : Total MapReduce CPU Time Spent: 0 msec
> 03:29:11 E   INFO  : Completed executing 
>

[jira] [Commented] (IMPALA-6532) NullPointerException in HiveContextAwareRecordReader.initIOContext() when executing Hive query

2018-04-27 Thread Joe McDonnell (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457215#comment-16457215
 ] 

Joe McDonnell commented on IMPALA-6532:
---

Dataload started doing parallel Hive execution. This is running into problems 
in 2.x based jobs (Hive-1.1.0 CDH5.16). I haven't seen it on master 
(Hive-2.1.1).

HiveContextAwareRecordReader.initIOContext() calls getIOContext()
{code:java}
private void initIOContext(long startPos, boolean isBlockPointer,
Path inputPath) {
  ioCxtRef = this.getIOContext();
  ioCxtRef.setCurrentBlockStart(startPos); // NullPointerException here
  ...
}{code}
getIOContext() calls IOContext.get(Configuration conf). This get() function 
signature is storing the IOContext in IOContext's inputNameIOContextMap. This 
map is not synchronized:
{code:java}
/**
 * Tez and MR use this map but are single threaded per JVM thus no 
synchronization is required.
 */
private static final Map inputNameIOContextMap = new 
HashMap();
{code}
I believe LocalJobRunner breaks this assumption and can have multiple threads 
accessing this, so this should not be expected to work.

Hive-2.1.1 contains a series of fixes adding synchronization to this structure:

HIVE-9460 - reworked the code and this map became thread local

HIVE-11146 - reword the code slightly (but doesn't seem to have modified the 
synchronization)

HIVE-11015 - refactored the map into IOContextMap. Now uses a synchronized 
global map.

> NullPointerException in HiveContextAwareRecordReader.initIOContext() when 
> executing Hive query
> --
>
> Key: IMPALA-6532
> URL: https://issues.apache.org/jira/browse/IMPALA-6532
> Project: IMPALA
>  Issue Type: Bug
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Bikramjeet Vig
>Assignee: Joe McDonnell
>Priority: Blocker
>  Labels: broken-build, flaky
>
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1273/console]
> {noformat}
> 02:48:03 [gw15] FAILED 
> metadata/test_partition_metadata.py::TestPartitionMetadata::test_partition_metadata_compatibility[exec_option:
>  {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
> {noformat}
> {noformat}
> 03:29:11 === FAILURES 
> ===
> 03:29:11  
> TestPartitionMetadata.test_partition_metadata_compatibility[exec_option: 
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
> 03:29:11 [gw15] linux2 -- Python 2.7.12 
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 03:29:11 metadata/test_partition_metadata.py:127: in 
> test_partition_metadata_compatibility
> 03:29:11 "insert into %s partition (x) values(1,1)" % FQ_TBL_HIVE)
> 03:29:11 common/impala_test_suite.py:684: in run_stmt_in_hive
> 03:29:11 raise RuntimeError(stderr)
> 03:29:11 E   RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
> 03:29:11 E   SLF4J: Found binding in 
> [jar:file:/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.15.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 03:29:11 E   SLF4J: Found binding in 
> [jar:file:/home/ubuntu/Impala/toolchain/cdh_components/hadoop-2.6.0-cdh5.15.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 03:29:11 E   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for 
> an explanation.
> 03:29:11 E   SLF4J: Actual binding is of type 
> [org.slf4j.impl.Log4jLoggerFactory]
> 03:29:11 E   scan complete in 3ms
> 03:29:11 E   Connecting to jdbc:hive2://localhost:11050
> 03:29:11 E   Connected to: Apache Hive (version 1.1.0-cdh5.15.0-SNAPSHOT)
> 03:29:11 E   Driver: Hive JDBC (version 1.1.0-cdh5.15.0-SNAPSHOT)
> 03:29:11 E   Transaction isolation: TRANSACTION_REPEATABLE_READ
> 03:29:11 E   No rows affected (0.045 seconds)
> 03:29:11 E   INFO  : Compiling 
> command(queryId=ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba): 
> insert into 
> test_partition_metadata_compatibility_b2ac5e.part_parquet_tbl_hive partition 
> (x) values(1,1)
> 03:29:11 E   INFO  : Semantic Analysis Completed
> 03:29:11 E   INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), 
> FieldSchema(name:_col1, type:string, comment:null)], properties:null)
> 03:29:11 E   INFO  : Completed compiling 
> command(queryId=ubuntu_20180215024848_d80982c5-a75c-4441-ab43-68d238eb69ba); 
> Time taken: 0.123 seconds
>

[jira] [Commented] (IMPALA-6340) There is no error when inserting an invalid value into a decimal column under decimal_v2

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457366#comment-16457366
 ] 

ASF subversion and git services commented on IMPALA-6340:
-

Commit d0f838b66a935310de9aeae7b9ba6513a5197ba9 in impala's branch 
refs/heads/master from [~tarasbob]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d0f838b ]

IMPALA-6340,IMPALA-6518: Check that decimal types are compatible in FE

In this patch we implement strict decimal type checking in the FE in
various situations when DECIMAL_V2 is enabled. What is affected:
- Union. If we union two decimals and it is not possible to come up
  with a decimal that will be able to contain all the digits, an error
  is thrown. For example, the union(decimal(20, 10), decimal(20, 20))
  returns decimal(30, 20). However, for union(decimal(38, 0),
  decimal(38, 38)) the ideal return type would be decimal(76,38), but
  this is too large, so an error is thrown.
- Insert. If we are inserting a decimal value into a column where we are
  not guaranteed that all digits will fit, an error is thrown. For
  example, inserting a decimal(38,0) value into a decimal(38,38) column.
- Functions such as coalesce(). If we are unable to determine the output
  type that guarantees that all digits will fit from all the arguments,
  an error is thrown. For example,
  coalesce(decimal(38,38), decimal(38,0)) will throw an error.
- Hash Join. When joining on two decimals, if a type cannot be
  determined that both columns can be cast to, we throw an error.
  For example, join on decimal(38,0) and decimal(38,38) will result
  in an error.

To avoid these errors, you need to use CAST() on some of the decimals.

In this patch we also change the output decimal calculation of decimal
round, truncate and related functions. If these functions are a no-op,
the resulting decimal type is the same as the input type.

Testing:
- Ran a core build which passed.

Change-Id: Id406f4189e01a909152985fabd5cca7a1527a568
Reviewed-on: http://gerrit.cloudera.org:8080/9930
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins 


> There is no error when inserting an invalid value into a decimal column under 
> decimal_v2
> 
>
> Key: IMPALA-6340
> URL: https://issues.apache.org/jira/browse/IMPALA-6340
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Taras Bobrovytsky
>Assignee: Taras Bobrovytsky
>Priority: Blocker
>  Labels: correctness
>
> The following series of commands does not result in an error or a warning 
> when decimal_v2 is enabled.
> {code}
> set decimal_v2=1;
> create table t1 (c1 decimal(38,37));
> insert into t1 select 11.11;
> {code}
> We end up inserting a NULL into the column without any warnings.
> If these commands are executed with decimal_v2 disabled, we get the following 
> warning:
> {code}
> WARNINGS: UDF WARNING: Decimal expression overflowed, returning NULL
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6314) Add run time scalar subquery check for uncorrelated subqueries

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457365#comment-16457365
 ] 

ASF subversion and git services commented on IMPALA-6314:
-

Commit 9c32594f722f139955788d0da57bdeecc3bacf75 in impala's branch 
refs/heads/2.x from [~boroknagyz]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=9c32594 ]

IMPALA-6934: Wrong results with EXISTS subquery containing ORDER BY, LIMIT, and 
OFFSET

Queries may return wrong results if an EXISTS subquery has
an ORDER BY with a LIMIT and OFFSET clause. The EXISTS
subquery may incorrectly evaluate to TRUE even though it is
FALSE.

The bug was found during the code review of IMPALA-6314
(https://gerrit.cloudera.org/#/c/9005/). Turned out
QueryStmt.setLimit() wipes the offset. I modified it to
keep the offset expr.

Added tests to 'PlannerTest/subquery-rewrite.test' and
'QueryTest/subquery.test'

Change-Id: I9693623d3d0a8446913261252f8e4a07935645e0
Reviewed-on: http://gerrit.cloudera.org:8080/10218
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins 


> Add run time scalar subquery check for uncorrelated subqueries
> --
>
> Key: IMPALA-6314
> URL: https://issues.apache.org/jira/browse/IMPALA-6314
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Backend, Frontend
>Reporter: Tim Armstrong
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>  Labels: planner, ramp-up, tpc-ds
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6934) Wrong results with EXISTS subquery containing ORDER BY, LIMIT, and OFFSET

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457364#comment-16457364
 ] 

ASF subversion and git services commented on IMPALA-6934:
-

Commit 9c32594f722f139955788d0da57bdeecc3bacf75 in impala's branch 
refs/heads/2.x from [~boroknagyz]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=9c32594 ]

IMPALA-6934: Wrong results with EXISTS subquery containing ORDER BY, LIMIT, and 
OFFSET

Queries may return wrong results if an EXISTS subquery has
an ORDER BY with a LIMIT and OFFSET clause. The EXISTS
subquery may incorrectly evaluate to TRUE even though it is
FALSE.

The bug was found during the code review of IMPALA-6314
(https://gerrit.cloudera.org/#/c/9005/). Turned out
QueryStmt.setLimit() wipes the offset. I modified it to
keep the offset expr.

Added tests to 'PlannerTest/subquery-rewrite.test' and
'QueryTest/subquery.test'

Change-Id: I9693623d3d0a8446913261252f8e4a07935645e0
Reviewed-on: http://gerrit.cloudera.org:8080/10218
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins 


> Wrong results with EXISTS subquery containing ORDER BY, LIMIT, and OFFSET
> -
>
> Key: IMPALA-6934
> URL: https://issues.apache.org/jira/browse/IMPALA-6934
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, 
> Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
>Reporter: Alexander Behm
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
>  Labels: correctness, planner
>
> Queries may return wrong results if an EXISTS subquery has an ORDER BY with a 
> LIMIT and OFFSET clause. The EXISTS subquery may incorrectly evaluate to TRUE 
> even though it s FALSE.
> Reproduction:
> {code}
> select count(*) from functional.alltypestiny t where
> exists (select id from functional.alltypestiny where id < 5 order by id limit 
> 10 offset 6);
> {code}
> The query should return "0" but it incorrectly returns "8" because an 
> incorrect plan without the offset is generated. See plan:
> {code}
> +-+
> | Explain String  |
> +-+
> | Max Per-Host Resource Reservation: Memory=0B|
> | Per-Host Resource Estimates: Memory=84.00MB |
> | Codegen disabled by planner |
> | |
> | PLAN-ROOT SINK  |
> | |   |
> | 08:AGGREGATE [FINALIZE] |
> | |  output: count:merge(*)   |
> | |   |
> | 07:EXCHANGE [UNPARTITIONED] |
> | |   |
> | 04:AGGREGATE|
> | |  output: count(*) |
> | |   |
> | 03:NESTED LOOP JOIN [LEFT SEMI JOIN, BROADCAST] |
> | |   |
> | |--06:EXCHANGE [BROADCAST]  |
> | |  ||
> | |  05:MERGING-EXCHANGE [UNPARTITIONED]  |
> | |  |  order by: id ASC  |
> | |  |  limit: 1  |
> | |  ||
> | |  02:TOP-N [LIMIT=1]   |
> | |  |  order by: id ASC  |
> | |  ||
> | |  01:SCAN HDFS [functional.alltypestiny]   |
> | | partitions=4/4 files=4 size=460B  |
> | | predicates: id < 5|
> | |   |
> | 00:SCAN HDFS [functional.alltypestiny t]|
> |partitions=4/4 files=4 size=460B |
> +-+
> {code}
> Evaluating the subquery by itself gives the expected results:
> {code}
> select id from functional.alltypestiny where id < 5 order by id limit 10 
> offset 6;
> 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-2831) Impala can spin up too many scanner threads

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457369#comment-16457369
 ] 

ASF subversion and git services commented on IMPALA-2831:
-

Commit 789c5aac23480acc6e18c057b767b65fdd791c97 in impala's branch 
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=789c5aa ]

IMPALA-6920: fix inconsistencies with scanner thread tokens

The first scanner thread to start now takes a "required" token,
which always succeeds. Only additional threads try to get
"optional" tokens, which can fail. Previously threads always
requested optional tokens, which could fail and leave the scan
node without any running threads until its callback is invoked.

This allows us to remove the "reserved optional token" and
set_max_quota() interfaces from ThreadResourceManager. There should
be no behavioural changes in ThreadResourceMgr in cases when those
features are not used.

Also switch Kudu to using the same logic for implementing
NUM_SCANNER_THREADS (it was not switched over to the improved
HDFS scanner logic added in IMPALA-2831).

Do some cleanup in ThreadResourceMgr code while we're here:
* Fix some benign data races in ThreadResourceMgr by switching to
  AtomicInt* classes.
* Remove pointless object caching (TCMalloc will do better).
* Reduce dependencies on the thread-resource-mgr.h header.

Testing:
Ran core tests.

Ran a few queries under TSAN, checked that it didn't report any more
races in this code after fixing those data races.

I couldn't construct a regression test because there are no easily
testable consequences of the change - the main difference is that
some scanner threads start earlier when there is pressure on scanner
thread tokens but that is hard to construct a robust test around.

Change-Id: I16d31d72441aff7293759281d0248e641df43704
Reviewed-on: http://gerrit.cloudera.org:8080/10186
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 


> Impala can spin up too many scanner threads
> ---
>
> Key: IMPALA-2831
> URL: https://issues.apache.org/jira/browse/IMPALA-2831
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.2, Impala 2.3.0, Impala 2.5.0
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Major
>  Labels: resource-management
> Fix For: Impala 2.7.0
>
>
> We have observed a number of problems with the way Impala dynamically creates 
> scanner threads, where more scanner threads are created than is ideal.
> * The scanner memory heuristic can lead to excessive memory consumption, 
> especially for very selective scans with wide rows. The current heuristic for 
> limiting memory consumption does not do well in these cases. There are likely 
> several interlinked causes here, which will need further investigation.
> * The non-deterministic scanner thread heuristic can lead to a great deal of 
> performance variability. At a minimum, the number of scanner threads should 
> always converge to the same number for the same plan and data if the query is 
> the only one running on the cluster.
> * Beyond a point, adding additional scanner threads does not improve 
> performance (and can degrade it), but the heuristic will keep on spinning up 
> scanner threads if there are tokens and memory available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6518) Fix the output type of a decimal union for decimal_v2

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457367#comment-16457367
 ] 

ASF subversion and git services commented on IMPALA-6518:
-

Commit d0f838b66a935310de9aeae7b9ba6513a5197ba9 in impala's branch 
refs/heads/master from [~tarasbob]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d0f838b ]

IMPALA-6340,IMPALA-6518: Check that decimal types are compatible in FE

In this patch we implement strict decimal type checking in the FE in
various situations when DECIMAL_V2 is enabled. What is affected:
- Union. If we union two decimals and it is not possible to come up
  with a decimal that will be able to contain all the digits, an error
  is thrown. For example, the union(decimal(20, 10), decimal(20, 20))
  returns decimal(30, 20). However, for union(decimal(38, 0),
  decimal(38, 38)) the ideal return type would be decimal(76,38), but
  this is too large, so an error is thrown.
- Insert. If we are inserting a decimal value into a column where we are
  not guaranteed that all digits will fit, an error is thrown. For
  example, inserting a decimal(38,0) value into a decimal(38,38) column.
- Functions such as coalesce(). If we are unable to determine the output
  type that guarantees that all digits will fit from all the arguments,
  an error is thrown. For example,
  coalesce(decimal(38,38), decimal(38,0)) will throw an error.
- Hash Join. When joining on two decimals, if a type cannot be
  determined that both columns can be cast to, we throw an error.
  For example, join on decimal(38,0) and decimal(38,38) will result
  in an error.

To avoid these errors, you need to use CAST() on some of the decimals.

In this patch we also change the output decimal calculation of decimal
round, truncate and related functions. If these functions are a no-op,
the resulting decimal type is the same as the input type.

Testing:
- Ran a core build which passed.

Change-Id: Id406f4189e01a909152985fabd5cca7a1527a568
Reviewed-on: http://gerrit.cloudera.org:8080/9930
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins 


> Fix the output type of a decimal union for decimal_v2
> -
>
> Key: IMPALA-6518
> URL: https://issues.apache.org/jira/browse/IMPALA-6518
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Taras Bobrovytsky
>Assignee: Taras Bobrovytsky
>Priority: Major
>
> In the following query, the type of c1 is decimal(38, 38) for both decimal v1 
> and v2.
> {code:java}
> select c1 from (
> select cast(1 as decimal(38, 0)) as c1
> union all
> select cast(0.1 as decimal(38, 38)) as c1) t{code}
> This means that we are truncating from the front. It would make more sense to 
> truncate from the back and round when decimal v2 is enabled. The output type 
> should be (38, 6). This is what we do for other mathematical operations, such 
> as addition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-6920) Multithreaded scans are not guaranteed to get a thread token immediately

2018-04-27 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/IMPALA-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457368#comment-16457368
 ] 

ASF subversion and git services commented on IMPALA-6920:
-

Commit 789c5aac23480acc6e18c057b767b65fdd791c97 in impala's branch 
refs/heads/master from [~tarmstr...@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=789c5aa ]

IMPALA-6920: fix inconsistencies with scanner thread tokens

The first scanner thread to start now takes a "required" token,
which always succeeds. Only additional threads try to get
"optional" tokens, which can fail. Previously threads always
requested optional tokens, which could fail and leave the scan
node without any running threads until its callback is invoked.

This allows us to remove the "reserved optional token" and
set_max_quota() interfaces from ThreadResourceManager. There should
be no behavioural changes in ThreadResourceMgr in cases when those
features are not used.

Also switch Kudu to using the same logic for implementing
NUM_SCANNER_THREADS (it was not switched over to the improved
HDFS scanner logic added in IMPALA-2831).

Do some cleanup in ThreadResourceMgr code while we're here:
* Fix some benign data races in ThreadResourceMgr by switching to
  AtomicInt* classes.
* Remove pointless object caching (TCMalloc will do better).
* Reduce dependencies on the thread-resource-mgr.h header.

Testing:
Ran core tests.

Ran a few queries under TSAN, checked that it didn't report any more
races in this code after fixing those data races.

I couldn't construct a regression test because there are no easily
testable consequences of the change - the main difference is that
some scanner threads start earlier when there is pressure on scanner
thread tokens but that is hard to construct a robust test around.

Change-Id: I16d31d72441aff7293759281d0248e641df43704
Reviewed-on: http://gerrit.cloudera.org:8080/10186
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 


> Multithreaded scans are not guaranteed to get a thread token immediately
> 
>
> Key: IMPALA-6920
> URL: https://issues.apache.org/jira/browse/IMPALA-6920
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: resource-management
>
> This bug applies to multithreaded HDFS and Kudu scans.
> So what happens is that we reserve an optional token for the first scanner 
> thread but that can be taken by any other operator in the same fragment. What 
> happens in one fragment in TPC-DS q18a is:
> 1. The hash join grabs an extra token for the join build. I guess it does 
> this early so it gets an optional token before other fragments can grab them.
> 2. The scan node reserves an optional token in Open(). This optional token is 
> already in use by the hash join.
> 3. The scan node tries to start the first scanner thread, but there are no 
> optional tokens available, so it can't start any.
> 4. Eventually the optional token is given up and the scanner thread can start.
> If #4 always happens without the scan making progress, then no deadlock is 
> possible, but if there's any kind of circular dependency, this can deadlock.
> Kudu scans also do not implement the num_scanner_threads query option in the 
> same way as HDFS scans - the IMPALA-2831 changes were not applied to kudu.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org