[jira] [Created] (IMPALA-6317) Expose -cmake_only flag to buildall.sh

2017-12-12 Thread David Knupp (JIRA) 
David Knupp created IMPALA-6317:
---

 Summary: Expose -cmake_only flag to buildall.sh
 Key: IMPALA-6317
 URL: https://issues.apache.org/jira/browse/IMPALA-6317
 Project: IMPALA
  Issue Type: Improvement
  Components: Infrastructure
Affects Versions: Impala 2.11.0
Reporter: David Knupp
Priority: Minor


Impala/bin/make_impala.sh has a {{-cmake_only}} command line option:
{noformat}
-cmake_only)
  CMAKE_ONLY=1
{noformat}

Passing this flag means that makefiles only will be generated during the build. 
However, this flag is not provided in buildall.sh (the caller of 
make_impala.sh) which effectively renders it useless.

It turns out that if one has no intention of running the Impala cluster locally 
(e.g., as when trying to build just enough of the toolchain and dev environment 
to run the data load scripts for loading data onto a remote cluster) then being 
able to only generate makefiles is a useful thing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-5881) Use native allocations to workaround JVM limitations in getAllCatalogObjects()

2017-12-12 Thread Alexander Behm (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Behm resolved IMPALA-5881.

Resolution: Won't Fix

We decided to fix this in a different way. See IMPALA-5990

> Use native allocations to workaround JVM limitations in getAllCatalogObjects()
> --
>
> Key: IMPALA-5881
> URL: https://issues.apache.org/jira/browse/IMPALA-5881
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Catalog
>Reporter: bharath v
>Assignee: bharath v
>
> When {{compact_catalog_topic=true}}, we need to compact the output of 
> getAllCatalogObjects() since it can potentially be quite big (on large 
> clusters) and we have the risk of running into JVM array limitations if we 
> use the regular TBinaryProtocol.Currently we only compact the output between 
> backend < - > statestore, but it makes sense to do it between backend< - 
> >frontend in the JNI too.
> Additionally, figure out ways to use native allocations to workaround JVM 
> array size limitations during thrift serialization.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-5934) Impala Thrift Server

2017-12-12 Thread Alexander Behm (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Behm resolved IMPALA-5934.

Resolution: Incomplete

> Impala Thrift Server 
> -
>
> Key: IMPALA-5934
> URL: https://issues.apache.org/jira/browse/IMPALA-5934
> Project: IMPALA
>  Issue Type: Bug
> Environment: Cloudera 5.10.1
>Reporter: Venkat Atmuri
> Fix For: Impala 2.7.0
>
>
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> E0914 10:58:10.457620 94112 logging.cc:121] stderr will be logged to this 
> file.
> W0914 10:58:10.467237 94112 authentication.cc:1003] LDAP authentication is 
> being used with TLS, but without an --ldap_ca_certificate file, the identity 
> of the LDAP server cannot be verified.  Network communication (and hence 
> passwords) could be intercepted by a man-in-the-middle attack
> E0914 10:58:13.220167 94268 thrift-server.cc:182] ThriftServer 'backend' (on 
> port: 22000) exited due to TException: Could not bind: Transport endpoint is 
> not connected
> E0914 10:58:13.220221 94112 thrift-server.cc:171] ThriftServer 'backend' (on 
> port: 22000) did not start correctly
> F0914 10:58:13.221709 94112 impalad-main.cc:89] ThriftServer 'backend' (on 
> port: 22000) did not start correctly
> . Impalad exiting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6315) Add run time scalar subquery check for correlated subqueries

2017-12-12 Thread Tim Armstrong (JIRA) 
Tim Armstrong created IMPALA-6315:
-

 Summary: Add run time scalar subquery check for correlated 
subqueries
 Key: IMPALA-6315
 URL: https://issues.apache.org/jira/browse/IMPALA-6315
 Project: IMPALA
  Issue Type: Sub-task
  Components: Backend, Frontend
Reporter: Tim Armstrong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6314) Add run time scalar subquery check for uncorrelated subqueries

2017-12-12 Thread Tim Armstrong (JIRA) 
Tim Armstrong created IMPALA-6314:
-

 Summary: Add run time scalar subquery check for uncorrelated 
subqueries
 Key: IMPALA-6314
 URL: https://issues.apache.org/jira/browse/IMPALA-6314
 Project: IMPALA
  Issue Type: Sub-task
  Components: Backend, Frontend
Reporter: Tim Armstrong
Assignee: Zoltán Borók-Nagy






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6313) length function returns incorrect value when input is a cast of a string literal

2017-12-12 Thread N Campbell (JIRA) 
N Campbell created IMPALA-6313:
--

 Summary: length function returns incorrect value when input is a 
cast of a string literal
 Key: IMPALA-6313
 URL: https://issues.apache.org/jira/browse/IMPALA-6313
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.9.0
Reporter: N Campbell
Priority: Minor



The following expressions return a value of 32

select length( concat( cast ( 'a' as char(32) ), '' ) )  from cert.tversion
select length( 'a   ') from cert.tversion

This expression returns  a value of 1 and not 32
select length( cast ( 'a' as char(32) ) ) from cert.tversion




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6308) RPC timeout message printing invalid destination name

2017-12-12 Thread Michael Ho (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Ho resolved IMPALA-6308.

Resolution: Fixed

https://github.com/apache/impala/commit/e57c77f095cbf1660b02d629f7b0cb6a92491e0e

IMPALA-6308: Fix bad Status() usage in data-stream-sender.cc

Previously, DataStreamSender::FlushAndSendEos() makes some wrong
assumption about the format string of the error status returned
from DoTransmitDataRpc(). In particular, it assumes that the format
string has exactly one substitution argument. This is a pretty bad
assumption as new types of errors could be returned from
DoTransmitDataRpc() and they can take different numbers of
substitution arguments. Recent changes in the format string
TErrorCode::RPC_RECV_TIMEOUT exposed this bug.

This change fixes the problem by removing this bad usage of Status()
in data-stream-sender.cc

Change-Id: I1cc04db670069a7df484e2f2bafb93c5315b9c82
Reviewed-on: http://gerrit.cloudera.org:8080/8817
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins

> RPC timeout message printing invalid destination name
> -
>
> Key: IMPALA-6308
> URL: https://issues.apache.org/jira/browse/IMPALA-6308
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Pranay Singh
>Assignee: Michael Ho
>Priority: Blocker
> Fix For: Impala 2.11.0
>
>
> Error message should print the destination address, when RPC timeout error is 
> encountered.
> Details
> --
> E1208 21:46:01.583712 29991 data-stream-sender.cc:278] channel send to 
> ec2-m2-4xlarge-centos-6-4-1e74.vpc.cloudera.com:22000 failed: RPC recv timed 
> out: dest address: ec2-m2-4xlarge-centos-6-4-1e74.vpc.cloudera.com:22000, 
> rpc: N6impala19TTransmitDataResultE
> F1208 21:46:18.957877 30189 substitute.cc:41] strings::Substitute format 
> string invalid: asked for "$1", but only 1 args were given.  Full format 
> string was: "RPC recv timed out: dest address: $0, rpc: $1".
> *** Check failure stack trace: ***
> @  0x3bdfc1d  google::LogMessage::Fail()
> @  0x3be14c2  google::LogMessage::SendToLog()
> @  0x3bdf5f7  google::LogMessage::Flush()
> @  0x3bdf959  google::LogMessage::~LogMessage()
> @  0x3bd63e9  strings::internal::SubstitutedSize()
> @  0x3bd67ec  strings::SubstituteAndAppend()
> @  0x15fbdde  strings::Substitute()
> @  0x1a3adf9  impala::ErrorMsg::ErrorMsg()
> @  0x1642c72  impala::Status::Status()
> @  0x2c88bbd  impala::DataStreamSender::Channel::FlushAndSendEos()
> @  0x2c8b648  impala::DataStreamSender::FlushFinal()
> @  0x189336c  impala::FragmentInstanceState::ExecInternal()
> @  0x189096f  impala::FragmentInstanceState::Exec()
> @  0x18797b8  impala::QueryState::ExecFInstance()
> @  0x187807a  
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @  0x187a3f7  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x17c6ed4  boost::function0<>::operator()()
> @  0x1abdbc9  impala::Thread::SuperviseThread()
> @  0x1ac6754  boost::_bi::list4<>::operator()<>()
> @  0x1ac6697  boost::_bi::bind_t<>::operator()()
> @  0x1ac665a  boost::detail::thread_data<>::run()
> @  0x2d6966a  thread_proxy
> @   0x31bee07851  (unknown)
> @   0x31beae894d  (unknown)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6242) Flaky test: TimerCounterTest.CountersTestOneThread

2017-12-12 Thread Tianyi Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tianyi Wang resolved IMPALA-6242.
-
Resolution: Fixed

IMPALA-6242: Reduce flakiness in TimerCounterTest

The error threshold in TimerCounterTest is 15ms, which is still not
enough in some rare cases. This patch increases it to 30ms.

Change-Id: Ifc038908857060ccbabfe30c46e72fd93907f412
Reviewed-on: http://gerrit.cloudera.org:8080/8670
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins


> Flaky test: TimerCounterTest.CountersTestOneThread
> --
>
> Key: IMPALA-6242
> URL: https://issues.apache.org/jira/browse/IMPALA-6242
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Tianyi Wang
>Assignee: Tianyi Wang
>Priority: Critical
>  Labels: broken-build
>
> CountersTestOneThread failed in an release build. The difference between the 
> time interval measured in spawned thread and that in main thread is more than 
> expected.
> {noformat}
> be/src/util/runtime-profile-test.cc:673
> Expected: (stopwatch_value) >= (expected_value - 
> TimerCounterTest::MAX_TIMER_ERROR_NS), actual: 250997548 vs 258086034
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6311) Evaluate smaller FPP for Bloom filters

2017-12-12 Thread Jim Apple (JIRA) 
Jim Apple created IMPALA-6311:
-

 Summary: Evaluate smaller FPP for Bloom filters
 Key: IMPALA-6311
 URL: https://issues.apache.org/jira/browse/IMPALA-6311
 Project: IMPALA
  Issue Type: Task
  Components: Perf Investigation
Reporter: Jim Apple


The Bloom filters are created by estimating the NDV and then using the FPP of 
75% to get the right size for the filter. This is may be too high to be very 
useful - if our filters are currently filtering more than 75% out, then it is 
only because we are overestimating NDV.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6310) Hung Query-During an exhaustive test run on CentOS6.5

2017-12-12 Thread yyzzjj (JIRA) 
yyzzjj created IMPALA-6310:
--

 Summary: Hung Query-During an exhaustive test run on CentOS6.5
 Key: IMPALA-6310
 URL: https://issues.apache.org/jira/browse/IMPALA-6310
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.11.0
Reporter: yyzzjj
Priority: Critical
 Attachments: mysql-san-node-part1.png, mysql-scan-node-part2.png, 
mysql-scanner.png


{code:java}
Thread 4 (Thread 0x7fd38ba69700 (LWP 19831)):
#0  0x003fbc60b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x7fe7b6b9a173 in boost::condition_variable::wait (this=0x7fddf2689258, 
m=...) at /usr/local/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x7fe7b2ed13fc in boost::thread::join_noexcept() () from 
/export/ldb/servers/impala-snapshot/lib_C/libboost_thread.so.1.57.0
#3  0x7fe7b66bda5f in boost::thread::join (this=0x31409aa0) at 
/usr/local/include/boost/thread/detail/thread.hpp:767
#4  impala::Thread::Join (this=) at 
/export/ldb/online/impala/be/src/util/thread.h:118
#5  impala::ThreadGroup::JoinAll (this=this@entry=0x7fde39ee4c38) at 
/export/ldb/online/impala/be/src/util/thread.cc:361
#6  0x7fe7b611549d in impala::MysqlScanNode::Close (this=0x7fde39ee4940, 
state=0x7fdfef3ac000) at 
/export/ldb/online/impala/be/src/jd_src/mysql-scan-node.cc:226
#7  0x7fe7b55cb2f9 in impala::ExecNode::Close (this=0x7fe522cb0f80, 
state=0x7fdfef3ac000) at /export/ldb/online/impala/be/src/exec/exec-node.cc:199
#8  0x7fe7b55cb2f9 in impala::ExecNode::Close (this=0x7fdff791a400, 
state=0x7fdfef3ac000) at /export/ldb/online/impala/be/src/exec/exec-node.cc:199
#9  0x7fe7b5c0e129 in impala::FragmentInstanceState::Close 
(this=this@entry=0x7fe16d6ef0e0) at 
/export/ldb/online/impala/be/src/runtime/fragment-instance-state.cc:301
#10 0x7fe7b5c113a9 in impala::FragmentInstanceState::Exec 
(this=this@entry=0x7fe16d6ef0e0) at 
/export/ldb/online/impala/be/src/runtime/fragment-instance-state.cc:95
#11 0x7fe7b5c2a628 in impala::QueryState::ExecFInstance (this=0xfe58400, 
fis=0x7fe16d6ef0e0) at 
/export/ldb/online/impala/be/src/runtime/query-state.cc:393
#12 0x7fe7b66c01c3 in boost::function0::operator() 
(this=0x7fd38ba68a40) at 
/usr/local/include/boost/function/function_template.hpp:767
#13 impala::Thread::SuperviseThread(std::string const&, std::string const&, 
boost::function, impala::Promise*) (name=..., category=..., 
functor=..., thread_started=0x7fd30d39beb0) at 
/export/ldb/online/impala/be/src/util/thread.cc:352
#14 0x7fe7b66c0a14 in 
boost::_bi::list4 >, 
boost::_bi::value >, boost::_bi::value >, 
boost::_bi::value >::operator()&, const std::basic_string&, 
boost::function, impala::Promise*), boost::_bi::list0> 
(f=@0x7fdec242bdb8: 0x7fe7b66c0010 , 
a=, this=0x7fdec242bdc0) at 
/usr/local/include/boost/bind/bind.hpp:457
#15 boost::_bi::bind_t, 
boost::_bi::value > >::operator()() 
(this=0x7fdec242bdb8) at /usr/local/include/boost/bind/bind_template.hpp:20
#16 boost::detail::thread_data, 
boost::_bi::value > > >::run() (this=0x7fdec242bc00) at 
/usr/local/include/boost/thread/detail/thread.hpp:116
#17 0x7fe7b2ed10f5 in thread_proxy () from 
/export/ldb/servers/impala-snapshot/lib_C/libboost_thread.so.1.57.0
#18 0x003fbc6079d1 in start_thread () from /lib64/libpthread.so.0
#19 0x003fbbee886d in clone () from /lib64/libc.so.6

{code}

{code:java}
Thread 3 (Thread 0x7fd31a5b2700 (LWP 20083)):
#0  0x003fbc60b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x7fe7b55d2fdb in impala::ConditionVariable::Wait (lock=..., 
this=) at 
/export/ldb/online/impala/be/src/util/condition-variable.h:41
#2  impala::BlockingQueue 
>::BlockingPut >(std::unique_ptr&&) (this=this@entry=0x7fdd41384e00, 
val=...) at /export/ldb/online/impala/be/src/util/blocking-queue.h:119
#3  0x7fe7b55ce8c5 in impala::ExecNode::RowBatchQueue::AddBatch 
(this=0x7fdd41384e00, batch=...) at 
/export/ldb/online/impala/be/src/exec/exec-node.cc:96