from:"Tim Armstrong \(JIRA\)"

[jira] [Resolved] (IMPALA-3396) New flaky test: TimerCounterTest.CountersTestRandom

2017-11-27 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-3396.
---
Resolution: Fixed

> New flaky test: TimerCounterTest.CountersTestRandom
> ---
>
> Key: IMPALA-3396
> URL: https://issues.apache.org/jira/browse/IMPALA-3396
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.6.0
>Reporter: Tim Armstrong
>Assignee: Juan Yu
>Priority: Blocker
>  Labels: broken-build
> Fix For: impala 2.5.1, Impala 2.6.0
>
>
> {code}
> [ RUN  ] TimerCounterTest.CountersTestRandom
> /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/runtime-profile-test.cc:615:
>  Failure
> Value of: stopwatch_value >= expected_value - 
> TimerCounterTest::MAX_TIMER_ERROR_NS && stopwatch_value <= expected_value + 
> TimerCounterTest::MAX_TIMER_ERROR_NS
>   Actual: false
> Expected: true
> /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/runtime-profile-test.cc:619:
>  Failure
> Value of: timer_value >= expected_value - 
> TimerCounterTest::MAX_TIMER_ERROR_NS && timer_value <= expected_value + 
> TimerCounterTest::MAX_TIMER_ERROR_NS
>   Actual: false
> Expected: true
> /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/runtime-profile-test.cc:615:
>  Failure
> Value of: stopwatch_value >= expected_value - 
> TimerCounterTest::MAX_TIMER_ERROR_NS && stopwatch_value <= expected_value + 
> TimerCounterTest::MAX_TIMER_ERROR_NS
>   Actual: false
> Expected: true
> /data/jenkins/workspace/impala-private-build-and-test/repos/Impala/be/src/util/runtime-profile-test.cc:619:
>  Failure
> Value of: timer_value >= expected_value - 
> TimerCounterTest::MAX_TIMER_ERROR_NS && timer_value <= expected_value + 
> TimerCounterTest::MAX_TIMER_ERROR_NS
>   Actual: false
> Expected: true
> [  FAILED  ] TimerCounterTest.CountersTestRandom (904 ms)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6249) Expose CMAKE_BUILD_TYPE via web UI for build type detection

2017-11-28 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6249:
-

 Summary: Expose CMAKE_BUILD_TYPE via web UI for build type 
detection
 Key: IMPALA-6249
 URL: https://issues.apache.org/jira/browse/IMPALA-6249
 Project: IMPALA
  Issue Type: Improvement
  Components: Infrastructure
Reporter: Tim Armstrong


IMPALA-6241 added a .cmake_build_type file with the CMAKE_BUILD_TYPE value for 
the last build. The file is used to detect the type of the build that the 
python tests are running against. However, this assumes that the tests are 
running from the same directory that the Impala cluster under test was built 
from, which isn't necessarily true for all dev workflows and for remote cluster 
tests.

It would be convenient if CMAKE_BUILD_TYPE was exposed from the Impalad web UI. 
Currently we expose DEBUG/RELEASE depending on the value of NDEBUG - see 
GetVersionString() and impalad-host:25000/?json=true, but we could expose the 
precise build type, then allow the python tests to parse it from the web UI.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-2887) improve Bitmap with a fast mod

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2887.
---
Resolution: Won't Fix

The code around this has changed a lot, unclear if this is still important.

> improve Bitmap with a fast mod
> --
>
> Key: IMPALA-2887
> URL: https://issues.apache.org/jira/browse/IMPALA-2887
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.3.0
>Reporter: Zuo Wang
>Priority: Minor
>  Labels: performance
>
> The probe filter is fairly expensive. check the profile below.
> This is due to the mod operator in Bitmap::Get.
> I will try to implement a fast mod according to "Division by Invariant 
> Integers Using Multiplication"(T. Granlund and P. L. Montgomery, 
> https://gmplib.org/~tege/divcnst-pldi94.pdf) 
> Function Stack Effective Time by Utilization Spin Time Overhead Time 
> Effective Time Spin Time Overhead Time Instructions Retired: Total 
> Instructions Retired: Self CPI Rate: Total CPI Rate: Self CPU Frequency 
> Ratio: Total CPU Frequency Ratio: Self Module Function (Full) Source File 
> Start Address
> Total 98.9% 1.1% 0.0% 0.0% 0s 0s 100.0% 0 0.496 0.000 1.080 0.000
> impala::HdfsParquetScanner::ScalarColumnReader (bool)1>::ReadNonRepeatedValue 42.7% 0.0% 0.0% 4.1% 0s 0s 66.0% 1,944,000,000 
> 0.329 1.041 1.108 1.081 impalad 
> impala::HdfsParquetScanner::ScalarColumnReader (bool)1>::ReadNonRepeatedValue(impala::MemPool*, impala::Tuple*, bool*) 
> hdfs-parquet-scanner.cc 0xbf1560
> ReadValue 38.6% 0.0% 0.0% 0.0% 0s 0s 64.1% 14,400,000 0.307 0.250 
> 1.111 impalad ReadValue hdfs-parquet-scanner.cc 0xbf156d
> ReadSlot 37.8% 0.0% 0.0% 7.4% 0s 0s 64.0% 4,989,600,000 0.302 0.768 
> 1.116 1.139 impalad ReadSlot hdfs-parquet-scanner.cc 0xbf15a2
> Get 26.6% 0.0% 0.0% 26.6% 0s 0s 52.4% 52,192,800,000 0.259 0.259 1.114 
> 1.114 impalad Get bitmap.h 0xbf15e2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-3553) LLVM CodeGen performance reduced

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-3553.
---
Resolution: Cannot Reproduce

> LLVM CodeGen performance reduced
> -
>
> Key: IMPALA-3553
> URL: https://issues.apache.org/jira/browse/IMPALA-3553
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.0, Impala 2.2
>Reporter: seven chen
>Priority: Minor
> Attachments: profile.jpg, report.rar
>
>
> when impala running about 3 months,the same sql will become slowly.
> According to the profile,i see the compile_timer_ take about 2m32s.
> The code detail is below.
>   SCOPED_TIMER(compile_timer_);
>   // JIT compile all codegen'd functions
>   for (int i = 0; i < fns_to_jit_compile_.size(); ++i) {
> *fns_to_jit_compile_[i].second = 
> JitFunction(fns_to_jit_compile_[i].first);
>   }
> If reboot the impala,CodeGen performance will normal.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-5543) Hide "advanced" query options from default list.

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-5543.
---
Resolution: Duplicate

This is being done as part of  IMPALA-2181

> Hide "advanced" query options from default list.
> 
>
> Key: IMPALA-5543
> URL: https://issues.apache.org/jira/browse/IMPALA-5543
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.10.0
>Reporter: Tim Armstrong
>  Labels: usability
>
> After IMPALA-2181 is fixed, we should re-evaluate the existing query options 
> and hide the ones that are not useful for more users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-5170) Streaming gzip decompression for Parquet and Avro files

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-5170.
---
Resolution: Won't Fix

> Streaming gzip decompression for Parquet and Avro files
> ---
>
> Key: IMPALA-5170
> URL: https://issues.apache.org/jira/browse/IMPALA-5170
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
>Reporter: Alexander Behm
>  Labels: ramp-up, usability
>
> To reduce the memory consumption of scans over gzip-compressed data, we 
> should implement streaming decompression (gzip supports it).
> Note that our text scanners already perform streaming decompression by 
> default, but that's not the case for other scanners (e.g. Parquet/Avro).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-3869) Perfromance down in KUDU as compare to HDFS

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-3869.
---
Resolution: Not A Bug

Hi Ravi, sorry you never got a response here. Usually we're better at 
responding to these type of questions on the u...@impala.apache.org list. We've 
done a lot of work on Kudu perf since you reported this so hopefully things are 
faster now.

> Perfromance down in KUDU as compare to HDFS
> ---
>
> Key: IMPALA-3869
> URL: https://issues.apache.org/jira/browse/IMPALA-3869
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Kudu_Impala
>Reporter: Ravi sharma
>Priority: Minor
> Attachments: query3.txt
>
>
> I am performing testing scenarios between IMPALA on HDFS vs IMPALA on KUDU
> we have set of queries which are accessing number of fact tables and 
> dimension tables.
> In one of the query we are trying to process 2 fact tables which are having 
> around 78 millions and 668 millions records.
> While having data in IMPALA on HDFS, i was able to get query results in less 
> than 50 seconds.
> But while having data in IMPALA on KUDU, even after trying number of 
> distributions/paritions, i have not been able to reduce query execution time 
> less than 125 seconds.
> So i have some conerns here...
> 1. In KUDU, what is the criteria of having number of cores/nodes in cluster 
> as per number of records to process...?
> 2. In KUDU, is there any option of like distributed cache in IMPALA on KUDU 
> to improve my execution time...?
> 3. Is there any other way to improve performance with having such huge data 
> load..?
> i have attached the query for reference..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-3590) impala crash at MemPool::FreeAll()

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-3590.
---
Resolution: Cannot Reproduce

> impala   crash  at MemPool::FreeAll()
> -
>
> Key: IMPALA-3590
> URL: https://issues.apache.org/jira/browse/IMPALA-3590
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.5.0
>Reporter: fishing
>Priority: Minor
> Attachments: hs_err_pid39974.log
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-5126) expr-test crashes on exit with boost::unordered::detail::functions<boost::hash

2017-11-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-5126.
---
Resolution: Cannot Reproduce

> expr-test crashes on exit with 
> boost::unordered::detail::functions
> Key: IMPALA-5126
> URL: https://issues.apache.org/jira/browse/IMPALA-5126
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Reporter: Zach Amsden
>Priority: Minor
>
> Trying to reproduce https://issues.apache.org/jira/browse/IMPALA-5112 I got 
> several instances of this core instead:
> {code}
> (gdb) bt
> #0  0x7fdee46f9c37 in __GI_raise (sig=sig@entry=6) at 
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x7fdee46fd028 in __GI_abort () at abort.c:89
> #2  0x7fdee46f2bf6 in __assert_fail_base (
> fmt=0x7fdee48433b8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
> assertion=assertion@entry=0x2fab2bf "!size_",
> file=file@entry=0x2fab268 
> "/home/impala/Impala/toolchain/boost-1.57.0-p1/include/boost/unordered/detail/table.hpp",
>  line=line@entry=567,
> function=function@entry=0x2fab640 
> ,
>  impala::RuntimeProfile::TimeSeriesCounter*, 
> boost::hash, 
> std::equal_to > 
> >::delete_buckets()::__PRETTY_FUNCTION__> "void 
> boost::unordered::detail::table::delete_buckets() [with Types = 
> boost::unordered::detail::set,
>  impala::RuntimeProfile::TimeSeries"...) at assert.c:92
> #3  0x7fdee46f2ca2 in __GI___assert_fail (assertion=0x2fab2bf "!size_",
> file=0x2fab268 
> "/home/impala/Impala/toolchain/boost-1.57.0-p1/include/boost/unordered/detail/table.hpp",
>  line=567,
> function=0x2fab640 
> ,
>  impala::RuntimeProfile::TimeSeriesCounter*, 
> boost::hash, 
> std::equal_to > 
> >::delete_buckets()::__PRETTY_FUNCTION__> "void 
> boost::unordered::detail::table::delete_buckets() [with Types = 
> boost::unordered::detail::set,
>  impala::RuntimeProfile::TimeSeries"...) at assert.c:101
> #4  0x0184ec05 in 
> boost::unordered::detail::table,
>  impala::RuntimeProfile::TimeSeriesCounter*, 
> boost::hash, 
> std::equal_to > >::delete_buckets 
> (this=0x41d7db8 )
> at 
> /home/impala/Impala/toolchain/boost-1.57.0-p1/include/boost/unordered/detail/table.hpp:567
> #5  0x0184d258 in 
> boost::unordered::detail::table,
>  impala::RuntimeProfile::TimeSeriesCounter*, 
> boost::hash, 
> std::equal_to > >::~table 
> (this=0x41d7db8 , 
> __in_chrg=)
> at 
> /home/impala/Impala/toolchain/boost-1.57.0-p1/include/boost/unordered/detail/table.hpp:520
> #6  0x0184c54c in 
> boost::unordered::detail::table_impl,
>  impala::RuntimeProfile::TimeSeriesCounter*, 
> boost::hash, 
> std::equal_to > >::~table_impl 
> (this=0x41d7db8 , 
> __in_chrg=)
> at 
> /home/impala/Impala/toolchain/boost-1.57.0-p1/include/boost/unordered/detail/unique.hpp:178
> #7  0x0184c5e4 in 
> boost::unordered::unordered_set boost::hash, 
> std::equal_to, 
> std::allocator >::~unordered_set (
> this=0x41d7db8 , 
> __in_chrg=)
> at 
> /home/impala/Impala/toolchain/boost-1.57.0-p1/include/boost/unordered/unordered_set.hpp:1040
> #8  0x01855728 in 
> impala::PeriodicCounterUpdater::~PeriodicCounterUpdater (
> this=0x41d7d00 , 
> __in_chrg=)
> at /home/impala/Impala/be/src/util/periodic-counter-updater.h:40
> #9

[jira] [Resolved] (IMPALA-2737) Investigate partition-oriented agg and join processing

2017-12-04 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2737.
---
Resolution: Later

Closing until we have more concrete plans.

> Investigate partition-oriented agg and join processing
> --
>
> Key: IMPALA-2737
> URL: https://issues.apache.org/jira/browse/IMPALA-2737
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.3.0
>Reporter: Tim Armstrong
>Priority: Minor
>  Labels: performance
> Attachments: partition-oriented-pagg-preview.diff
>
>
> Currently the partitioned aggregations and joins add rows to the partitions 
> as they process the input. This leads to poor memory access patterns since 
> the 16 different partitions are randomly accessed. An alternative approach is 
> to do an initial pass to hash and divide the rows between partitions, then do 
> a second pass per partition to insert all the rows for that partition. This 
> avoids the random access to partitions.
> This can enable some additional optimisations, e.g. prefetching hash table 
> buckets for the next row.
> An initial prototype was posted here: http://gerrit.cloudera.org/#/c/628 . 
> The diff is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6324) Support reading RLE-encoded boolean values in Parquet scanner

2017-12-14 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6324:
-

 Summary: Support reading RLE-encoded boolean values in Parquet 
scanner
 Key: IMPALA-6324
 URL: https://issues.apache.org/jira/browse/IMPALA-6324
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Reporter: Tim Armstrong


Per this discussion on the Parquet mailing list, RLE will become a valid 
encoding for the boolean type in parquet. We should add support for reading 
this.

https://mail-archives.apache.org/mod_mbox/parquet-dev/201712.mbox/%3CCAJPUwMDbGgkS1WmN8OvuuA%3DQ%2BXd%2BOwLn2XZAu7CNGF1sMVZMJg%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6185) disk-io-mgr backend test gets stuck on RHEL7

2017-12-13 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6185.
---
Resolution: Not A Bug

This is an issue with a particular kernel version.

> disk-io-mgr backend test gets stuck on RHEL7
> 
>
> Key: IMPALA-6185
> URL: https://issues.apache.org/jira/browse/IMPALA-6185
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Taras Bobrovytsky
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: flaky
> Attachments: gdb.txt
>
>
> The test is stuck for 24 hours on RHEL7 before the build is aborted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6313) length function returns incorrect value when input is a cast of a string literal

2017-12-13 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6313.
---
Resolution: Duplicate

Looks like we already are tracking this. I'm not sure what the use-case is but 
my suggestion would be to use STRING or VARCHAR in preference to CHAR. Even 
without this bug, the correct behaviour of CHAR is often counter-intuitive 
because of the rules about when the padding spaces are added and removed.

> length function returns incorrect value when input is a cast of a string 
> literal
> 
>
> Key: IMPALA-6313
> URL: https://issues.apache.org/jira/browse/IMPALA-6313
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.9.0, Impala 2.11.0
>Reporter: N Campbell
>Priority: Critical
>  Labels: correctness
>
> The following expressions return a value of 32
> {noformat}
> select length( concat( cast ( 'a' as char(32) ), '' ) )  from cert.tversion
> select length( 'a   ') from cert.tversion
> {noformat}
> This expression returns  a value of 1 and not 32
> {noformat}
> select length( cast ( 'a' as char(32) ) ) from cert.tversion
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6314) Add run time scalar subquery check for uncorrelated subqueries

2017-12-12 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6314:
-

 Summary: Add run time scalar subquery check for uncorrelated 
subqueries
 Key: IMPALA-6314
 URL: https://issues.apache.org/jira/browse/IMPALA-6314
 Project: IMPALA
  Issue Type: Sub-task
  Components: Backend, Frontend
Reporter: Tim Armstrong
Assignee: Zoltán Borók-Nagy






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6315) Add run time scalar subquery check for correlated subqueries

2017-12-12 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6315:
-

 Summary: Add run time scalar subquery check for correlated 
subqueries
 Key: IMPALA-6315
 URL: https://issues.apache.org/jira/browse/IMPALA-6315
 Project: IMPALA
  Issue Type: Sub-task
  Components: Backend, Frontend
Reporter: Tim Armstrong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-1575) Cancelled queries do not yield resources until close

2017-11-20 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1575.
---
   Resolution: Fixed
Fix Version/s: Impala 2.11.0

IMPALA-1575: part 2: yield admission control resources

This change releases admission control resources more eagerly,
once the query has finished actively executing. Some resources
(tracked and untracked) are still consumed by the client request
as long as it remains open, e.g. memory for control structures
and the result cache. However, these resources are relatively
small and should not block admission of new queries.

The same as in part 1, query execution is considered to be finished
under any of the following conditions:
1. The query encounters an error and fails
2. The query is cancelled due to the idle query timeout
3. The query reaches eos (or the DML completes)
4. The client cancels the query without closing the query

Admission control resources are released in two ways:
1. by calling AdmissionController::ReleaseQuery() on the coordinator
   promptly after query execution finishes, instead of waiting for
   UnregisterQuery(). This means that the query and its memory is
   no longer considered "admitted".
2. by changing the behaviour of MemTracker::GetPoolMemReserved() so
   that it is aware of when a query has finished executing and does not
   consider its entire memory limit to be "reserved".

The preconditions for releasing an admitted query are subtle because the
queries are being admitted to a distributed system, not just the
coordinator.  The comment for ReleaseAdmissionControlResources()
documents the preconditions and rationale. Note that the preconditions
are not weaker than the preconditions of calling UnregisterQuery()
before this patch.

Testing:
TestAdmissionController is extended to end queries in four ways:
cancellation by client, idle timeout, the last row being fetched,
and the client closing the query. The test uses a mix of all four.
After the query ends, all clients wait for the test to complete
before closing the query or closing the connection. This ensures
that the admission control decisions are based entirely on the
query end behavior. This test works for both query admission control
and mem_limit admission control and can detect both kinds of admission
control resources ("admitted" and "reserved") not being released
promptly.

I ran into a problem similar to IMPALA-3772 with the admission control
tests becoming flaky due to query timeouts on release builds, which I
solved in a similar way by increasing the frequency of statestore
updates.

This is based on an earlier patch by Joe McDonnell.

Change-Id: Ib1fae8dc1c4b0eca7bfa8fadae4a56ef2b37947a
Reviewed-on: http://gerrit.cloudera.org:8080/8581
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins


> Cancelled queries do not yield resources until close
> 
>
> Key: IMPALA-1575
> URL: https://issues.apache.org/jira/browse/IMPALA-1575
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.1, Impala 2.3.0
>Reporter: Henry Robinson
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: query-lifecycle, resource-management
> Fix For: Impala 2.11.0
>
>
> A cancelled query (for example due to a timeout) or a query that has reached 
> eos (but not explicitly closed) holds (1) resources on the coordinator 
> fragment, (2) all resources accounted by the admission controller, (3) llama 
> reservations. (However, Llama has been unsupported for CDH 5.5 and beyond, so 
> (3) will no longer apply.) All of these are not released until the query is 
> closed, which may not happen promptly for some clients.
> This frequently occurs with Hue. Hue (and some other clients that behave 
> similarly) will not close a query until explicitly closed (in the Hue case 
> this is via a javascript callback sent by the browser when closing the Hue 
> tab). If the query is left unattended (or the Hue tab is on a laptop that is 
> closed, or the browser crashes), the close call is never sent, and while the 
> query will "time out", the cancellation doesn't properly clean up resources.
> One way to mitigate this issue in this case is by using the 
> --idle_session_timeout impalad argument to fully close a session and all 
> associated queries after some amount of time (but this is not a workaround 
> that works in all cases).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6224) impala-shell exits with error "No closing quotation" with certain syntactic errors

2017-11-20 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6224:
-

 Summary: impala-shell exits with error "No closing quotation" with 
certain syntactic errors
 Key: IMPALA-6224
 URL: https://issues.apache.org/jira/browse/IMPALA-6224
 Project: IMPALA
  Issue Type: Bug
  Components: Clients
Reporter: Tim Armstrong


{code}
with v as (select 1)
select foo('\\'), ('bar
;
{code}

Pasting the above into impala-shell results in the shell exiting with an error. 
It should print a syntax error but the shell should not exit:
{noformat}
[localhost:21000] > with v as (select 1)
  > select foo('\\'), ('bar
  > ;
Traceback (most recent call last):
  File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line 
1541, in 
shell.cmdloop(intro)
  File "/usr/lib/python2.7/cmd.py", line 142, in cmdloop
stop = self.onecmd(line)
  File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line 
558, in onecmd
return cmd.Cmd.onecmd(self, line)
  File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
return func(arg)
  File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line 
1087, in do_with
tokens = list(lexer)
  File "/usr/lib/python2.7/shlex.py", line 269, in next
token = self.get_token()
  File "/usr/lib/python2.7/shlex.py", line 96, in get_token
raw = self.read_token()
  File "/usr/lib/python2.7/shlex.py", line 172, in read_token
raise ValueError, "No closing quotation"
ValueError: No closing quotation
{noformat}

The repro query is a reduced version of a more complex query reported by a 
customer.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6224) impala-shell exits with error "No closing quotation" with certain syntactic errors

2017-11-20 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6224.
---
Resolution: Fixed

> impala-shell exits with error "No closing quotation" with certain syntactic 
> errors
> --
>
> Key: IMPALA-6224
> URL: https://issues.apache.org/jira/browse/IMPALA-6224
> Project: IMPALA
>  Issue Type: Bug
>  Components: Clients
>Reporter: Tim Armstrong
>  Labels: usability
>
> {code}
> with v as (select 1)
> select foo('\\'), ('bar
> ;
> {code}
> Pasting the above into impala-shell results in the shell exiting with an 
> error. It should print a syntax error but the shell should not exit:
> {noformat}
> [localhost:21000] > with v as (select 1)
>   > select foo('\\'), ('bar
>   > ;
> Traceback (most recent call last):
>   File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line 
> 1541, in 
> shell.cmdloop(intro)
>   File "/usr/lib/python2.7/cmd.py", line 142, in cmdloop
> stop = self.onecmd(line)
>   File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line 
> 558, in onecmd
> return cmd.Cmd.onecmd(self, line)
>   File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
> return func(arg)
>   File "/home/tarmstrong/Impala/incubator-impala/shell/impala_shell.py", line 
> 1087, in do_with
> tokens = list(lexer)
>   File "/usr/lib/python2.7/shlex.py", line 269, in next
> token = self.get_token()
>   File "/usr/lib/python2.7/shlex.py", line 96, in get_token
> raw = self.read_token()
>   File "/usr/lib/python2.7/shlex.py", line 172, in read_token
> raise ValueError, "No closing quotation"
> ValueError: No closing quotation
> {noformat}
> The repro query is a reduced version of a more complex query reported by a 
> customer.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6222) Make it easier to root-cause "failed to get minimum memory reservation" error

2017-11-20 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6222:
-

 Summary: Make it easier to root-cause "failed to get minimum 
memory reservation" error
 Key: IMPALA-6222
 URL: https://issues.apache.org/jira/browse/IMPALA-6222
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Affects Versions: Impala 2.10.0
Reporter: Tim Armstrong
Assignee: Bikramjeet Vig


A user reported this error message:

{noformat}
 ExecQueryFInstances rpc query_id=c94288312d6d4055:bbfa1665 failed: 
Failed to get minimum memory reservation of 26.69 MB on daemon 
hodor-030.edh.cloudera.com:22000 for query c94288312d6d4055:bbfa1665 
because it would exceed an applicable memory limit. Memory is likely 
oversubscribed. Reducing query concurrency or configuring admission control may 
help avoid this error. Memory usage:
Process: Limit=96.00 GB Total=16.54 GB Peak=83.37 GB
{noformat}

It turns out that a query was using up a lot of reservation, but it wasn't 
immediately apparent that the process reservation was mostly allocated to that 
query.
{noformat}
Process: Limit=96.00 GB Total=12.20 GB Peak=83.37 GB
  Buffer Pool: Free Buffers: Total=208.00 MB
  Buffer Pool: Clean Pages: Total=7.19 GB
  Buffer Pool: Unused Reservation: Total=-76.79 GB
  Free Disk IO Buffers: Total=1.37 GB Peak=1.37 GB
  RequestPool=root.default: Total=76.81 GB Peak=77.56 GB
Query(464a9afdbf2646cf:d9e2d411): Reservation=76.80 GB 
ReservationLimit=76.80 GB OtherMemory=6.69 MB Total=76.81 GB Peak=76.93 GB
  Fragment 464a9afdbf2646cf:d9e2d411003f: Reservation=76.80 GB 
OtherMemory=6.69 MB Total=76.81 GB Peak=76.81 GB
SELECT_NODE (id=3): Total=20.00 KB Peak=9.02 MB
  Exprs: Total=4.00 KB Peak=4.00 KB
ANALYTIC_EVAL_NODE (id=2): Reservation=4.00 MB OtherMemory=6.64 MB 
Total=10.64 MB Peak=15.04 MB
  Exprs: Total=4.00 KB Peak=4.00 KB
SORT_NODE (id=1): Reservation=76.79 GB OtherMemory=16.00 KB Total=76.79 
GB Peak=76.80 GB
EXCHANGE_NODE (id=4): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=10.19 MB
DataStreamSender (dst_id=5): Total=1.48 KB Peak=1.48 KB
CodeGen: Total=1.68 KB Peak=710.00 KB
  Fragment 464a9afdbf2646cf:d9e2d4110016: Reservation=0 OtherMemory=0 
Total=0 Peak=389.69 MB
HDFS_SCAN_NODE (id=0): Total=0 Peak=388.54 MB
DataStreamSender (dst_id=4): Total=0 Peak=1.23 MB
CodeGen: Total=0 Peak=49.00 KB
  Untracked Memory: Total=3.42 GB
{noformat}

When a user or admin sees this problem they really want to immediately know:
* What resource is exhausted (i.e. the process-wide reservation)?
* Which query(ies) are using it and how do I kill them (i.e. what are the query 
ids and coordinators of the query).

We should think through the error messages and diagnostics and improve them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6226) Analytic planner misses some opportunities to merge sort groups

2017-11-20 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6226:
-

 Summary: Analytic planner misses some opportunities to merge sort 
groups
 Key: IMPALA-6226
 URL: https://issues.apache.org/jira/browse/IMPALA-6226
 Project: IMPALA
  Issue Type: Improvement
  Components: Frontend
Affects Versions: Impala 2.10.0
Reporter: Tim Armstrong
Priority: Minor


{noformat}
create table testrn (g int, v int, r int);
insert into testrn (g,v,r) select 1,1,1 union select 1,2,2 union select 2,1,3 
union select 2,2,4;
select g,first_value(v) over(partition by g order by r),row_number() over 
(partition by g order by r desc) from testrn;
{noformat}

The above analytical function query produces the following plan:
{noformat}
Max Per-Host Resource Reservation: Memory=20.00MB
Per-Host Resource Estimates: Memory=52.00MB
WARNING: The following tables are missing relevant table and/or column 
statistics.
default.testrn

F02:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=0B mem-reservation=0B
PLAN-ROOT SINK
|  mem-estimate=0B mem-reservation=0B
|
06:EXCHANGE [UNPARTITIONED]
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=6,3 row-size=24B cardinality=unavailable
|
F01:PLAN FRAGMENT [HASH(g)] hosts=1 instances=1
Per-Host Resources: mem-estimate=20.00MB mem-reservation=20.00MB
04:ANALYTIC
|  functions: row_number()
|  partition by: g
|  order by: r DESC
|  window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|  mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
|  tuple-ids=6,3 row-size=24B cardinality=unavailable
|
03:SORT
|  order by: g ASC NULLS FIRST, r DESC
|  mem-estimate=6.00MB mem-reservation=6.00MB spill-buffer=2.00MB
|  tuple-ids=6 row-size=16B cardinality=unavailable
|
02:ANALYTIC
|  functions: first_value(v)
|  partition by: g
|  order by: r ASC
|  window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|  mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
|  tuple-ids=4,2 row-size=16B cardinality=unavailable
|
01:SORT
|  order by: g ASC NULLS FIRST, r ASC
|  mem-estimate=6.00MB mem-reservation=6.00MB spill-buffer=2.00MB
|  tuple-ids=4 row-size=12B cardinality=unavailable
|
05:EXCHANGE [HASH(g)]
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=0 row-size=12B cardinality=unavailable
|
F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1
Per-Host Resources: mem-estimate=32.00MB mem-reservation=0B
00:SCAN HDFS [default.testrn, RANDOM]
   partitions=1/1 files=1 size=24B
   stats-rows=unavailable extrapolated-rows=disabled
   table stats: rows=unavailable size=unavailable
   column stats: unavailable
   mem-estimate=32.00MB mem-reservation=0B
   tuple-ids=0 row-size=12B cardinality=unavailable
{noformat}

I believe we could avoid sorting twice by rewriting first_value() into 
last_value(). I suspect there are a few other cases where the analytic 
functions can be "flipped" like this to handle a different sort order.

The solution probably involves adding some extra cases to this logic: 
https://github.com/apache/incubator-impala/blob/5e9b4e2/fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java#L729
 to detect when the expressions are identical up to the ASC/DESC and when the 
function can be "flipped" to produce the correct result.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6039) BitReader::GetAligned() doesn't zero out trailing bytes

2017-11-16 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6039.
---
   Resolution: Fixed
Fix Version/s: Impala 2.11.0

IMPALA-4177,IMPALA-6039: batched bit reading and rle decoding

Switch the decoders to using more batch-oriented interfaces. As an
intermediate step this doesn't make the interfaces of LevelDecoder
or DictDecoder batch-oriented, only the lower-level utility classes.

The next step would be to change those interfaces to be batch-oriented
and make according optimisations in parquet. This could deliver much
larger perf improvements than the current patch.

The high-level changes are.
* BitReader -> BatchedBitReader, which is built to unpack runs of 32
  bit-packed values efficiently.
* RleDecoder -> RleBatchDecoder, which exposes the repeated and literal
  runs to the caller and uses BatchedBitReader to unpack literal runs
  efficiently.
* Dict decoding uses RleBatchDecoder to decode repeated runs efficiently
  and uses the BitPacking utilities to unpack and encode in a single
  step.

Also removes an older benchmark that isn't too interesting (since
the batch-oriented approach to encoding and decoding is so much
faster than the value-by-value approach).

Testing:
* Ran core tests.
* Updated unit tests to exercise new code.
* Added test coverage for the deprecated bit-packed level encoding to
  that it still works (there was no coverage previously).

Perf:
Single-node benchmarks showed a few % performance gain. 16 node cluster
benchmarks only showed a gain for TPC-H nested.

Change-Id: I35de0cf80c86f501c4a39270afc8fb8111552ac6
Reviewed-on: http://gerrit.cloudera.org:8080/8267
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins
---

> BitReader::GetAligned() doesn't zero out trailing bytes
> ---
>
> Key: IMPALA-6039
> URL: https://issues.apache.org/jira/browse/IMPALA-6039
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Minor
> Fix For: Impala 2.11.0
>
>
> BitReader::GetAligned() only sets the initial bytes of the output value, 
> leaving the remaining bytes set to whatever they were previously. It isn't 
> clear if this is intentional and undocumented or a latent bug.
> The problem is non-obvious because the current callsites either call it with 
> a single byte at a time (GetVlqInt()), or initialize the output value to zero 
> and always call GetAligned() with the same num_bytes value (RleDecoder).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6296) DCheck in CodegenSymbolEmitter when --asm_module_dir is set on debug build

2017-12-07 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6296:
-

 Summary: DCheck in CodegenSymbolEmitter when --asm_module_dir is 
set on debug build
 Key: IMPALA-6296
 URL: https://issues.apache.org/jira/browse/IMPALA-6296
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.10.0, Impala 2.9.0, Impala 2.8.0, Impala 2.11.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


{noformat}
F1208 00:53:44.045367 18484 codegen-symbol-emitter.cc:90] Check failed: 
perf_map_.find(obj.getData().data()) != perf_map_.end()
{noformat}

To reproduce, run with --asm_module_dir=some_dir, where some_dir exists and do 
not set --perf_map=true. Any codegen'd query will hit the DCHECK.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6275) Successful CTAS logs warning

2017-12-05 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6275:
-

 Summary: Successful CTAS logs warning
 Key: IMPALA-6275
 URL: https://issues.apache.org/jira/browse/IMPALA-6275
 Project: IMPALA
  Issue Type: Bug
  Components: Frontend
Affects Versions: Impala 2.11.0
Reporter: Tim Armstrong


I just noticed this on my development branch. I'm not sure if it's new 
behaviour, but it's potentially confusing. I ran a successful CTAS for a table 
that didn't exist and got a stack trace in the warning log.

{noformat}
I1205 10:42:21.316118 14901 Frontend.java:909] Compiling query: create table 
new_table as select "foo"
W1205 10:42:21.320310 14901 Table.java:187] Could not load column statistics 
for: default.new_table
Java exception follows:
NoSuchObjectException(message:Specified database/table does not exist : 
default.new_table)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:3117)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:3104)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1557)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:101)
at com.sun.proxy.$Proxy13.getTableColumnStatistics(Unknown Source)
at org.apache.impala.catalog.Table.loadAllColumnStats(Table.java:185)
at org.apache.impala.catalog.HdfsTable.loadSchema(HdfsTable.java:1527)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1182)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1146)
at 
org.apache.impala.analysis.CreateTableAsSelectStmt.analyze(CreateTableAsSelectStmt.java:201)
at 
org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:388)
at 
org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:369)
at org.apache.impala.service.Frontend.analyzeStmt(Frontend.java:920)
at 
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1069)
at 
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
I1205 10:42:21.320755 14901 Frontend.java:948] Compiled query.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6291) Various crashes and incorrect results on CPUs with AVX512

2017-12-11 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6291.
---
Resolution: Fixed

I ran exhaustive tests in release and exhaustive - looks good. I also tested a 
few different branches, all of which passed exhaustive tests in release and 
debug:
* LLVM 3.9.1 (master)
* LLVM 3.8.0 (CDH5.13.1, derived from Impala 2.10)
* LLVM 5.0 ([~bikram.sngh91]'s work in progress branch)

> Various crashes and incorrect results on CPUs with AVX512
> -
>
> Key: IMPALA-6291
> URL: https://issues.apache.org/jira/browse/IMPALA-6291
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, 
> Impala 2.10.0, Impala 2.11.0
> Environment: Ubuntu 16.04, M5.4xlarge
>Reporter: Jim Apple
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: correctness, crash
> Fix For: Impala 2.11.0
>
>
> M5 and C5 instances use a different hypervisor than M4 and C4. In EC2 C5 and 
> M5 instances, data loading fails. An interesting snippet from the end of an 
> impalad log:
> {noformat}
> I1207 04:12:07.922456 19933 coordinator.cc:99] Exec() 
> query_id=944ead2f178cf67e:1755131f stmt=CREATE TABLE 
> tmp_orders_string AS 
>   SELECT STRAIGHT_JOIN
> o_orderkey, o_custkey, o_orderstatus, o_totalprice, o_orderdate,
> o_orderpriority, o_clerk, o_shippriority, o_comment,
> GROUP_CONCAT(
>   CONCAT(
> CAST(l_partkey AS STRING), '\005',
> CAST(l_suppkey AS STRING), '\005',
> CAST(l_linenumber AS STRING), '\005',
> CAST(l_quantity AS STRING), '\005',
> CAST(l_extendedprice AS STRING), '\005',
> CAST(l_discount AS STRING), '\005',
> CAST(l_tax AS STRING), '\005',
> CAST(l_returnflag AS STRING), '\005',
> CAST(l_linestatus AS STRING), '\005',
> CAST(l_shipdate AS STRING), '\005',
> CAST(l_commitdate AS STRING), '\005',
> CAST(l_receiptdate AS STRING), '\005',
> CAST(l_shipinstruct AS STRING), '\005',
> CAST(l_shipmode AS STRING), '\005',
> CAST(l_comment AS STRING)
>   ), '\004'
> ) AS lineitems_string
>   FROM tpch_parquet.lineitem
>   INNER JOIN [SHUFFLE] tpch_parquet.orders ON o_orderkey = l_orderkey
>   WHERE o_orderkey % 1 = 0
>   GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9
> ...
> F1207 04:12:08.972215 19953 partitioned-hash-join-node.cc:291] Check failed: 
> probe_batch_pos_ == probe_batch_->num_rows() || probe_batch_pos_ == -1 
> {noformat}
> The error log shows:
> {noformat}
> F1207 04:12:08.972215 19953 partitioned-hash-join-node.cc:291] Check failed: 
> probe_batch_pos_ == probe_batch_->num_rows() || probe_batch_pos_ == -1 
> *** Check failure stack trace: ***
> @  0x3bdcefd  google::LogMessage::Fail()
> @  0x3bde7a2  google::LogMessage::SendToLog()
> @  0x3bdc8d7  google::LogMessage::Flush()
> @  0x3bdfe9e  google::LogMessageFatal::~LogMessageFatal()
> @  0x28bd4db  impala::PartitionedHashJoinNode::NextProbeRowBatch()
> @  0x28c1741  impala::PartitionedHashJoinNode::GetNext()
> @  0x289f71f  
> impala::PartitionedAggregationNode::GetRowsStreaming()
> @  0x289d8d5  impala::PartitionedAggregationNode::GetNext()
> @  0x1891d1c  impala::FragmentInstanceState::ExecInternal()
> @  0x188f629  impala::FragmentInstanceState::Exec()
> @  0x1878c0a  impala::QueryState::ExecFInstance()
> @  0x18774cc  
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @  0x1879849  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x17c64ba  boost::function0<>::operator()()
> @  0x1abb5a1  impala::Thread::SuperviseThread()
> @  0x1ac412c  boost::_bi::list4<>::operator()<>()
> @  0x1ac406f  boost::_bi::bind_t<>::operator()()
> @  0x1ac4032  boost::detail::thread_data<>::run()
> @  0x2d668ca  thread_proxy
> @ 0x7fe9287146ba  start_thread
> @ 0x7fe92844a3dd  clone
> Picked up JAVA_TOOL_OPTIONS: 
> -agentlib:jdwp=transport=dt_socket,address=30002,server=y,suspend=n 
> {noformat}
> To reproduce this, start a M5.4xlarge with 250GB space
> {noformat}
> sudo apt-get update
> sudo apt-get install --yes git
> git init ~/Impala
> pushd ~/Impala
> git fetch https://github.com/apache/impala master
> git

[jira] [Created] (IMPALA-6287) Support ViewFS (HDFS Federation)

2017-12-06 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6287:
-

 Summary: Support ViewFS (HDFS Federation)
 Key: IMPALA-6287
 URL: https://issues.apache.org/jira/browse/IMPALA-6287
 Project: IMPALA
  Issue Type: Improvement
  Components: Frontend
Affects Versions: Impala 2.10.0
Reporter: Tim Armstrong


We should support viewfs:// filesystems in Impala. Currently they are not 
supported - see IMPALA-77.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6299) IRBuilder codegen should using LLVMCodeGen::cpu_attrs instead of CpuInfo to determine valid instructions

2017-12-08 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6299:
-

 Summary: IRBuilder codegen should using LLVMCodeGen::cpu_attrs 
instead of CpuInfo to determine valid instructions
 Key: IMPALA-6299
 URL: https://issues.apache.org/jira/browse/IMPALA-6299
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Reporter: Tim Armstrong


This came out of a discussion on this code review: 
https://gerrit.cloudera.org/#/c/8802/

The idea is that IRBuilder code that emits intrinsics should only do so if we 
have those CPU features enabled in LLVM. Otherwise codegen will fail to compile 
the emitted instructions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-6291) Various crashes and incorrect results on CPUs with AVX512

2017-12-08 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6291.
---
   Resolution: Fixed
Fix Version/s: Impala 2.11.0

IMPALA-6291: disable AVX512 codegen in LLVM

Adds a whitelist of LLVM CPU attributes that I know that we routinely
test Impala with. This excludes the problematic AVX512 attributes as
well as some other flags we don't test with - e.g. AMD-only
instructions, NVM-related instructions, etc. We're unlikely to get
significant benefit from these instruction set extensions without
explicitly using them via instrinsics.

Testing:
Ran core tests on a system with AVX512 support with a prototype patch
that disabled only the AVX512 flags. Added a backend test to make sure
that the whitelisting is working as expected.

Change-Id: Ic7c3ee3e370bafc50d855113485b7e6925f7bf6a
Reviewed-on: http://gerrit.cloudera.org:8080/8802
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins

> Various crashes and incorrect results on CPUs with AVX512
> -
>
> Key: IMPALA-6291
> URL: https://issues.apache.org/jira/browse/IMPALA-6291
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, 
> Impala 2.10.0, Impala 2.11.0
> Environment: Ubuntu 16.04, M5.4xlarge
>Reporter: Jim Apple
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: correctness, crash
> Fix For: Impala 2.11.0
>
>
> M5 and C5 instances use a different hypervisor than M4 and C4. In EC2 C5 and 
> M5 instances, data loading fails. An interesting snippet from the end of an 
> impalad log:
> {noformat}
> I1207 04:12:07.922456 19933 coordinator.cc:99] Exec() 
> query_id=944ead2f178cf67e:1755131f stmt=CREATE TABLE 
> tmp_orders_string AS 
>   SELECT STRAIGHT_JOIN
> o_orderkey, o_custkey, o_orderstatus, o_totalprice, o_orderdate,
> o_orderpriority, o_clerk, o_shippriority, o_comment,
> GROUP_CONCAT(
>   CONCAT(
> CAST(l_partkey AS STRING), '\005',
> CAST(l_suppkey AS STRING), '\005',
> CAST(l_linenumber AS STRING), '\005',
> CAST(l_quantity AS STRING), '\005',
> CAST(l_extendedprice AS STRING), '\005',
> CAST(l_discount AS STRING), '\005',
> CAST(l_tax AS STRING), '\005',
> CAST(l_returnflag AS STRING), '\005',
> CAST(l_linestatus AS STRING), '\005',
> CAST(l_shipdate AS STRING), '\005',
> CAST(l_commitdate AS STRING), '\005',
> CAST(l_receiptdate AS STRING), '\005',
> CAST(l_shipinstruct AS STRING), '\005',
> CAST(l_shipmode AS STRING), '\005',
> CAST(l_comment AS STRING)
>   ), '\004'
> ) AS lineitems_string
>   FROM tpch_parquet.lineitem
>   INNER JOIN [SHUFFLE] tpch_parquet.orders ON o_orderkey = l_orderkey
>   WHERE o_orderkey % 1 = 0
>   GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9
> ...
> F1207 04:12:08.972215 19953 partitioned-hash-join-node.cc:291] Check failed: 
> probe_batch_pos_ == probe_batch_->num_rows() || probe_batch_pos_ == -1 
> {noformat}
> The error log shows:
> {noformat}
> F1207 04:12:08.972215 19953 partitioned-hash-join-node.cc:291] Check failed: 
> probe_batch_pos_ == probe_batch_->num_rows() || probe_batch_pos_ == -1 
> *** Check failure stack trace: ***
> @  0x3bdcefd  google::LogMessage::Fail()
> @  0x3bde7a2  google::LogMessage::SendToLog()
> @  0x3bdc8d7  google::LogMessage::Flush()
> @  0x3bdfe9e  google::LogMessageFatal::~LogMessageFatal()
> @  0x28bd4db  impala::PartitionedHashJoinNode::NextProbeRowBatch()
> @  0x28c1741  impala::PartitionedHashJoinNode::GetNext()
> @  0x289f71f  
> impala::PartitionedAggregationNode::GetRowsStreaming()
> @  0x289d8d5  impala::PartitionedAggregationNode::GetNext()
> @  0x1891d1c  impala::FragmentInstanceState::ExecInternal()
> @  0x188f629  impala::FragmentInstanceState::Exec()
> @  0x1878c0a  impala::QueryState::ExecFInstance()
> @  0x18774cc  
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @  0x1879849  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x17c64ba  boost::function0<>::operator()()
> @  0x1abb5a1  impala::Thread::SuperviseThread()
> @  0x1ac412c  boost::_bi::list4<>::operator()<>()
> @  0x1ac406f

[jira] [Resolved] (IMPALA-6968) TestBlockVerificationGcmDisabled failure in exhaustive

2018-05-04 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6968.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> TestBlockVerificationGcmDisabled failure in exhaustive
> --
>
> Key: IMPALA-6968
> URL: https://issues.apache.org/jira/browse/IMPALA-6968
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {code:java}
> /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-release-thrift/repos/Impala/be/src/runtime/tmp-file-mgr-test.cc:550
> Value of: read_status.code()
>   Actual: 0
> Expected: TErrorCode::SCRATCH_READ_VERIFY_FAILED
> Which is: 118
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6979) BloomFilterBenchmark hits DCHECK

2018-05-04 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6979:
-

 Summary: BloomFilterBenchmark hits DCHECK
 Key: IMPALA-6979
 URL: https://issues.apache.org/jira/browse/IMPALA-6979
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Reporter: Tim Armstrong


Leaving this here in case someone else runs into it and needs to fix the 
benchmark. We don't run this benchmark as part of builds so it's not a high 
priority to fix.

{noformat}
F0504 15:18:55.533821 26709 bloom-filter.cc:192] Check failed: 
!out->always_false 
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6977) Report CPU usage (user/sys) per fragment

2018-05-04 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6977:
-

 Summary: Report CPU usage (user/sys) per fragment
 Key: IMPALA-6977
 URL: https://issues.apache.org/jira/browse/IMPALA-6977
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Reporter: Tim Armstrong


The exec summary only shows wallclock time, because that is the only time 
reported per operator.
{noformat}
Operator   #Hosts  Avg Time  Max Time  #Rows  Est. #Rows   Peak Mem  Est. 
Peak Mem  Detail 
---
03:AGGREGATE1   1.782ms   1.782ms  1   1   28.16 KB   
10.00 MB  FINALIZE   
02:EXCHANGE 1  90.073us  90.073us  3   1  104.00 KB 
 0  UNPARTITIONED  
01:AGGREGATE3   1s560ms   1s657ms  3   11.35 MB   
10.00 MB 
00:SCAN HDFS3   1s567ms   1s815ms  6.00M   6.00M   59.11 MB  
144.00 MB  tpch_seq_gzip.lineitem 
{noformat}

This is fairly useful for understanding query bottlenecks but is misleading whn 
it comes to determining where CPU was consumed. We have accurate User and Sys 
time for fragment instances so we could have a CPU usage summary based on 
profile information
{noformat}
Fragment   #Hosts  Avg UserTime  Max UserTime  Avg SysTime   Max SysTime
--
F011   12.000ms  ...   ..
F003   7s245ms   7s624ms   ...   ..
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6227) TestAdmissionControllerStress can be flaky

2018-05-08 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6227.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

I'm going to close since I don't have any next steps and the last commit solved 
one of the root causes of the flakiness (inconsistent metrics).

> TestAdmissionControllerStress can be flaky
> --
>
> Key: IMPALA-6227
> URL: https://issues.apache.org/jira/browse/IMPALA-6227
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Csaba Ringhofer
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments: TEST-impala-custom-cluster.xml
>
>
> jenkins build https://jenkins.impala.io/job/gerrit-verify-dryrun/1503/console 
> failed at the following test:
> {noformat}
> 01:30:11 ] === FAILURES 
> ===
> 01:30:11 ]  TestAdmissionControllerStress.test_mem_limit[num_queries: 30 | 
> submission_delay_ms: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: 
> text/none | round_robin_submission: True] 
> 01:30:11 ] custom_cluster/test_admission_controller.py:877: in test_mem_limit
> 01:30:11 ] {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> 01:30:11 ] custom_cluster/test_admission_controller.py:760: in 
> run_admission_test
> 01:30:11 ] assert metric_deltas['rejected'] ==\
> 01:30:11 ] E   assert 5 == ((30 - 15) - 15)
> {noformat}
> This is probably related to the following  recent commit:
> https://github.com/apache/incubator-impala/commit/7487c5de04c2c5d97b8a8d5c935d10568f1ed686



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-605) HBASE_CACHING default incorrectly displayed

2018-05-09 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-605.
--
Resolution: Duplicate

> HBASE_CACHING default incorrectly displayed
> ---
>
> Key: IMPALA-605
> URL: https://issues.apache.org/jira/browse/IMPALA-605
> Project: IMPALA
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: Impala 1.1.1
>Reporter: Ricky Saltzer
>Priority: Minor
>  Labels: shell
>
> In the impala-shell, the default value of _HBASE_CACHING_ is shown as *0*, 
> when in fact it's *1000* (I think). 
> {code}
> [my-impala-daemon:21000] > SET;
> Default query options:
>   ABORT_ON_DEFAULT_LIMIT_EXCEEDED: 0
>   ABORT_ON_ERROR: 0
>   ALLOW_UNSUPPORTED_FORMATS: 0
>   BATCH_SIZE: 0
>   DEBUG_ACTION: 
>   DEFAULT_ORDER_BY_LIMIT: -1
>   DISABLE_CODEGEN: 0
>   HBASE_CACHE_BLOCKS: 0
>   HBASE_CACHING: 0  <--
>   MAX_ERRORS: 0
>   MAX_IO_BUFFERS: 0
>   MAX_SCAN_RANGE_LENGTH: 0
>   MEM_LIMIT: 0
>   NUM_NODES: 0
>   NUM_SCANNER_THREADS: 0
>   PARQUET_COMPRESSION_CODEC: SNAPPY
>   SUPPORT_START_OVER: false
> {code}
> This can be pretty confusing since it's implicitly using a different value. 
> In my case, it was reading my HBase table which has *very* wide rows, and it 
> was causing the region servers to fall over.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6946) Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue

2018-04-27 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-6946:
-

 Summary: Hit DCHECK in impala::RleBatchDecoder::GetRepeatedValue
 Key: IMPALA-6946
 URL: https://issues.apache.org/jira/browse/IMPALA-6946
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 2.13.0, 
Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


The bug comes from conversion between signed and unsigned.

{noformat}
 DCHECK_GT(num_repeats_to_consume, 0);

(gdb) p num_repeats_to_consume
$2 = -1003251240
{noformat}

{noformat}
#2  0x01f422f0 in impala::ScalarColumnReader::ReadNonRepeatedValueBatch (this=0x6f547d50, 
pool=0x7fb37bdbb330, max_values=0, 
tuple_size=90, tuple_mem=0x0, num_values=0x7fb37bdbb330) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
#3  0x03fceea7 in google::LogMessage::Flush() ()
#4  0x03fd246e in google::LogMessageFatal::~LogMessageFatal() ()
#5  0x01f97197 in impala::RleBatchDecoder::GetRepeatedValue (this=0x1803fc6a0, num_repeats_to_consume=-1003251240)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/rle-encoding.h:493
#6  0x01f96d7b in 
impala::DictDecoder::DecodeNextValue (this=0x1803fc698, 
value=0x16b7e9000)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:455
#7  0x01f94528 in 
impala::DictDecoder::GetNextValue (value=0x16b7e9000, 
this=0x1803fc698)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:446
#8  impala::ScalarColumnReader::ReadSlot (pool=0x6f547d98, tuple=0x16b7e9000, 
this=0x1803fc400)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:499
#9  impala::ScalarColumnReader::MaterializeValueBatch (this=0x1803fc400, 
pool=0x6f547d98, max_values=1, tuple_size=90, 
tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:413
#10 0x01f67b4a in impala::ScalarColumnReader::MaterializeValueBatch 
(this=0x1803fc400, pool=0x6f547d98, max_values=1, 
tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:436
#11 0x01f657c2 in impala::ScalarColumnReader::ReadValueBatch (this=0x1803fc400, 
pool=0x6f547d98, max_values=1, tuple_size=90, 
tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:354
#12 0x01f422f0 in impala::ScalarColumnReader::ReadNonRepeatedValueBatch (this=0x1803fc400, 
pool=0x6f547d98, max_values=1, 
tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
#13 0x01d70a1c in impala::HdfsParquetScanner::AssembleRows 
(this=0x1a926000, column_readers=..., row_batch=0xdfc0780, 
skip_row_group=0x1a9261b8)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:936
#14 0x01d6d4f7 in impala::HdfsParquetScanner::GetNextInternal 
(this=0x1a926000, row_batch=0xdfc0780)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:434
#15 0x01d6b6c6 in impala::HdfsParquetScanner::ProcessSplit 
(this=0x1a926000) at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:332
#16 0x01cf30e4 in impala::HdfsScanNode::ProcessSplit (this=0x10c21000, 
filter_ctxs=..., expr_results_pool=0x7fb37bdbc480, scan_range=0xf3276c0, 
scanner_thread_reservation=0x7fb37bdbc400)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:482
#17 0x01cf2476 in impala::HdfsScanNode::ScannerThread (this=0x10c21000, 
scanner_thread_reservation=90112)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:385
#18 0x01cf18a8 in impala::HdfsScanNode::::operator()(void) 
const (__closure=0x7fb37bdbcbc8)
at 
/data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:300

[jira] [Resolved] (IMPALA-6945) IMPALA-6892 doesn't clean pick to 2.x

2018-04-27 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6945.
---
   Resolution: Fixed
Fix Version/s: Impala 2.13.0

> IMPALA-6892 doesn't clean pick to 2.x
> -
>
> Key: IMPALA-6945
> URL: https://issues.apache.org/jira/browse/IMPALA-6945
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Michael Brown
>Assignee: Tim Armstrong
>Priority: Blocker
> Fix For: Impala 2.13.0
>
>
> {noformat}
> mikeb@mikeb-ub16:1:~/Impala (2.x|CHERRY-PICKING) $ git diff
> diff --cc be/src/exec/exec-node.cc
> index c94f38c,d2dd85f..000
> --- a/be/src/exec/exec-node.cc
> +++ b/be/src/exec/exec-node.cc
> @@@ -68,11 -68,6 +68,14 @@@
>   using strings::Substitute;
> ++<<< HEAD
>  +DECLARE_int32(be_port);
>  +DECLARE_string(hostname);
>  +DEFINE_bool_hidden(enable_partitioned_hash_join, true, "Deprecated - has no 
> effect");
>  +DEFINE_bool_hidden(enable_partitioned_aggregation, true, "Deprecated - has 
> no effect");
>  +
> ++===
> ++>>> 518bcd3... IMPALA-6892: CheckHashAndDecrypt() includes file and host
>   namespace impala {
>   const string ExecNode::ROW_THROUGHPUT_COUNTER = "RowsReturnedRate";
> mikeb@mikeb-ub16:0:~/Impala (2.x|CHERRY-PICKING) $
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7013) test_insert_permutation failed with "timed out waiting for receiver fragment instance"

2018-05-10 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7013:
-

 Summary: test_insert_permutation failed with "timed out waiting 
for receiver fragment instance"
 Key: IMPALA-7013
 URL: https://issues.apache.org/jira/browse/IMPALA-7013
 Project: IMPALA
  Issue Type: Bug
  Components: Distributed Exec
Affects Versions: Impala 2.13.0
Reporter: Tim Armstrong
Assignee: Joe McDonnell


This is a weird failure that I don't quite understand

{noformat}
11:32:57  TestInsertQueriesWithPermutation.test_insert_permutation[exec_option: 
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 
'exec_single_node_rows_threshold': 0} | table_format: text/none] 
11:32:57 [gw2] linux2 -- Python 2.6.6 
/data/jenkins/workspace/impala-cdh5-trunk-core-s3/repos/Impala/bin/../infra/python/env/bin/python
11:32:57 query_test/test_insert_permutation.py:50: in test_insert_permutation
11:32:57 self.run_test_case('QueryTest/insert_permutation', vector)
11:32:57 common/impala_test_suite.py:405: in run_test_case
11:32:57 result = self.__execute_query(target_impalad_client, query, 
user=user)
11:32:57 common/impala_test_suite.py:620: in __execute_query
11:32:57 return impalad_client.execute(query, user=user)
11:32:57 common/impala_connection.py:160: in execute
11:32:57 return self.__beeswax_client.execute(sql_stmt, user=user)
11:32:57 beeswax/impala_beeswax.py:173: in execute
11:32:57 handle = self.__execute_query(query_string.strip(), user=user)
11:32:57 beeswax/impala_beeswax.py:341: in __execute_query
11:32:57 self.wait_for_completion(handle)
11:32:57 beeswax/impala_beeswax.py:361: in wait_for_completion
11:32:57 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
11:32:57 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
11:32:57 EQuery aborted:Sender 127.0.0.1 timed out waiting for receiver 
fragment instance: 994ce8cbd8cb4eb2:f1cb807c, dest node: 1
11:32:57  Captured stderr setup 
-
11:32:57 -- connecting to: localhost:21000
11:32:57 - Captured stderr call 
-
11:32:57 -- executing against localhost:21000
11:32:57 use default;
11:32:57 
11:32:57 SET sync_ddl=1;
11:32:57 -- executing against localhost:21000
11:32:57 drop database if exists `insert_permutation_test` cascade;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 use functional;
11:32:57 
11:32:57 SET batch_size=0;
11:32:57 SET num_nodes=0;
11:32:57 SET disable_codegen_rows_threshold=0;
11:32:57 SET disable_codegen=False;
11:32:57 SET abort_on_error=1;
11:32:57 SET exec_single_node_rows_threshold=0;
11:32:57 -- executing against localhost:21000
11:32:57 create database insert_permutation_test location
11:32:57 's3a://impala-cdh5-s3-test/test-warehouse/insert_permutation_test';
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 use insert_permutation_test;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 create table perm_nopart(int_col1 int, string_col string, int_col2 
int);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 
11:32:57 create table perm_part(int_col1 int, string_col string) partitioned by 
(p1 int, p2 string);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 
11:32:57 create table parquet_part(int_col1 int, string_col string)
11:32:57 partitioned by (p1 int, p2 string) stored as parquet;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart(int_col1, string_col, int_col2) 
values(1,'str',2);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart(int_col2, string_col, int_col1) 
values(1,'str',2);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_nopart(int_col2) values(1);
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 select * from perm_nopart;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 invalidate metadata  insert_permutation_test.perm_part;
11:32:57 
11:32:57 -- executing against localhost:21000
11:32:57 insert into perm_part(p1, string_col, int_col1, p2) values(10,'str',1, 
'hello');
11:32:57 
11:32:57 -- executing against localhost:21000

[jira] [Created] (IMPALA-7014) Disable stacktrace symbolisation by default

2018-05-11 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7014:
-

 Summary: Disable stacktrace symbolisation by default
 Key: IMPALA-7014
 URL: https://issues.apache.org/jira/browse/IMPALA-7014
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Affects Versions: Not Applicable
Reporter: Tim Armstrong
Assignee: Joe McDonnell


We got burned by the code of producing stacktrace again with IMPALA-6996. I did 
a quick investigation into this, based on the hypothesis that the symbolisation 
was the expensive part, rather than getting the addresses. I added a stopwatch 
to GetStackTrace() to measure the time in nanoseconds and ran a test that 
produces a backtrace

The first experiment was 
{noformat}
$ start-impala-cluster.py --impalad_args='--symbolize_stacktrace=true' && 
impala-py.test tests/query_test/test_scanners.py -k codec

I0511 09:45:11.897944 30904 debug-util.cc:283] stacktrace time: 75175573
I0511 09:45:11.897956 30904 status.cc:125] File 
'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_308108.db/bad_codec/bad_codec.parquet'
 uses an unsupported compression: 5000 for column 'id'.
@  0x18782ef  impala::Status::Status()
@  0x2cbe96f  impala::ParquetMetadataUtils::ValidateRowGroupColumn()
@  0x205f597  impala::BaseScalarColumnReader::Reset()
@  0x1feebe6  impala::HdfsParquetScanner::InitScalarColumns()
@  0x1fe6ff3  impala::HdfsParquetScanner::NextRowGroup()
@  0x1fe58d8  impala::HdfsParquetScanner::GetNextInternal()
@  0x1fe3eea  impala::HdfsParquetScanner::ProcessSplit()
@  0x1f6ba36  impala::HdfsScanNode::ProcessSplit()
@  0x1f6adc4  impala::HdfsScanNode::ScannerThread()
@  0x1f6a1c4  
_ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
@  0x1f6c2a6  
_ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
@  0x1bd3b1a  boost::function0<>::operator()()
@  0x1ebecd5  impala::Thread::SuperviseThread()
@  0x1ec6e71  boost::_bi::list5<>::operator()<>()
@  0x1ec6d95  boost::_bi::bind_t<>::operator()()
@  0x1ec6d58  boost::detail::thread_data<>::run()
@  0x31b3ada  thread_proxy
@ 0x7f9be67d36ba  start_thread
@ 0x7f9be650941d  clone

{noformat}
The stacktrace took 75ms, which is pretty bad! It would be worse on a 
production system with more memory maps.

The next experiment was to disable it:
{noformat}
start-impala-cluster.py --impalad_args='--symbolize_stacktrace=false' && 
impala-py.test tests/query_test/test_scanners.py -k codec

I0511 09:43:47.574185 29514 debug-util.cc:283] stacktrace time: 29528
I0511 09:43:47.574193 29514 status.cc:125] File 
'hdfs://localhost:20500/test-warehouse/test_bad_compression_codec_cb5d0225.db/bad_codec/bad_codec.parquet'
 uses an unsupported compression: 5000 for column 'id'.
@  0x18782ef
@  0x2cbe96f
@  0x205f597
@  0x1feebe6
@  0x1fe6ff3
@  0x1fe58d8
@  0x1fe3eea
@  0x1f6ba36
@  0x1f6adc4
@  0x1f6a1c4
@  0x1f6c2a6
@  0x1bd3b1a
@  0x1ebecd5
@  0x1ec6e71
@  0x1ec6d95
@  0x1ec6d58
@  0x31b3ada
@ 0x7fbdcbdef6ba
@ 0x7fbdcbb2541d
{noformat}
That's 2545x faster! If the addresses are in the statically linked binary, we 
can use addrline to get back the line numbers:
{noformat}
$ addr2line -e be/build/latest/service/impalad 0x2cbe96f
/home/tarmstrong/Impala/incubator-impala/be/src/exec/parquet-metadata-utils.cc:166
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7012) NullPointerException with CTAS query

2018-05-10 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7012:
-

 Summary: NullPointerException with CTAS query
 Key: IMPALA-7012
 URL: https://issues.apache.org/jira/browse/IMPALA-7012
 Project: IMPALA
  Issue Type: Bug
  Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tianyi Wang


{noformat}
[localhost:21000] default> create table alltypesinsert partitioned by (year, 
month) stored as parquet as /* +noclustered */select at1.id, at1.bool_col, 
at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col, 
 
at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, 
at1.timestamp_col,   
  at1.year, at2.id as month
from  functional.alltypes at1, functional.alltypes at2;
Query: create table alltypesinsert partitioned by (year, month) stored as 
parquet as /* +noclustered */
select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col, 
at1.bigint_col,
  at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, 
at1.timestamp_col,
  at1.year, at2.id as month
from  functional.alltypes at1, functional.alltypes at2
Query submitted at: 2018-05-10 13:46:02 (Coordinator: 
http://tarmstrong-box:25000)
ERROR: NullPointerException: null

{noformat}

{noformat}
I0510 13:46:02.977249  4238 Frontend.java:987] Analyzing query: create table 
alltypesinsert partitioned by (year, month) stored as parquet as /* 
+noclustered */
select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col, 
at1.bigint_col,
  at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, 
at1.timestamp_col,
  at1.year, at2.id as month
from  functional.alltypes at1, functional.alltypes at2
I0510 13:46:03.025013  4238 jni-util.cc:230] java.lang.NullPointerException
at org.apache.impala.analysis.SqlScanner.isReserved(SqlScanner.java:725)
at org.apache.impala.analysis.SqlParser.getErrorMsg(SqlParser.java:1532)
at org.apache.impala.service.Frontend.parse(Frontend.java:975)
at 
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:990)
at 
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
I0510 13:46:03.124739  4238 status.cc:125] NullPointerException: null
@  0x18782ef  impala::Status::Status()
@  0x1e55652  impala::JniUtil::GetJniExceptionMsg()
@  0x1d133ed  impala::JniUtil::CallJniMethod<>()
@  0x1d10047  impala::Frontend::GetExecRequest()
@  0x1d3205a  impala::ImpalaServer::ExecuteInternal()
@  0x1d31ba2  impala::ImpalaServer::Execute()
@  0x1d9be70  impala::ImpalaServer::query()
@  0x2ee378e  beeswax::BeeswaxServiceProcessor::process_query()
@  0x2ee34dc  beeswax::BeeswaxServiceProcessor::dispatchCall()
@  0x2ebcf9d  impala::ImpalaServiceProcessor::dispatchCall()
@  0x1836690  apache::thrift::TDispatchProcessor::process()
@  0x1b9649d  
apache::thrift::server::TAcceptQueueServer::Task::run()
@  0x1b8d9c5  impala::ThriftThread::RunRunnable()
@  0x1b8f0c9  boost::_mfi::mf2<>::operator()()
@  0x1b8ef5f  boost::_bi::list3<>::operator()<>()
@  0x1b8ecab  boost::_bi::bind_t<>::operator()()
@  0x1b8ebbe  
boost::detail::function::void_function_obj_invoker0<>::invoke()
@  0x1bd3b1a  boost::function0<>::operator()()
@  0x1ebec51  impala::Thread::SuperviseThread()
@  0x1ec6ded  boost::_bi::list5<>::operator()<>()
@  0x1ec6d11  boost::_bi::bind_t<>::operator()()
@  0x1ec6cd4  boost::detail::thread_data<>::run()
@  0x31b3a4a  thread_proxy
@ 0x7fcf12d536ba  start_thread
@ 0x7fcf12a8941d  clone
I0510 13:46:03.124944  4238 impala-server.cc:1010] UnregisterQuery(): 
query_id=6b4791bb7a54de54:16bdcba7
I0510 13:46:03.124948  4238 impala-server.cc:1097] Cancel(): 
query_id=6b4791bb7a54de54:16bdcba7
{noformat}

This is on commit hash 3e736450354e55244e16924cfeb223a30629351d  . It looks 
like the code was added by IMPALA-3916



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6970) DiskMgr::AllocateBuffersForRange crashes on failed DCHECK

2018-05-08 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6970.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> DiskMgr::AllocateBuffersForRange crashes on failed DCHECK
> -
>
> Key: IMPALA-6970
> URL: https://issues.apache.org/jira/browse/IMPALA-6970
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0
>Reporter: Sailesh Mukil
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments: stacks.txt
>
>
> Similar to IMPALA-6587, but the DCHECK failed in a slightly different way. 
> Cannot tell if the root cause is the same as that though without further 
> investigation.
> {code:java}
> FSF0503 09:30:26.715791 30750 reservation-tracker.cc:376] Check failed: bytes 
> <= unused_reservation() (8388608 vs. 6291456) 
> *** Check failure stack trace: ***
> @  0x4277c1d  google::LogMessage::Fail()
> @  0x42794c2  google::LogMessage::SendToLog()
> @  0x42775f7  google::LogMessage::Flush()
> @  0x427abbe  google::LogMessageFatal::~LogMessageFatal()
> @  0x1ef1343  impala::ReservationTracker::AllocateFromLocked()
> @  0x1ef111d  impala::ReservationTracker::AllocateFrom()
> @  0x1ee8c57  
> impala::BufferPool::Client::PrepareToAllocateBuffer()
> @  0x1ee5543  impala::BufferPool::AllocateBuffer()
> @  0x2f50f68  impala::io::DiskIoMgr::AllocateBuffersForRange()
> @  0x1f74762  impala::HdfsScanNodeBase::StartNextScanRange()
> @  0x1f6b052  impala::HdfsScanNode::ScannerThread()
> @  0x1f6a4ea  
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @  0x1f6c5cc  
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @  0x1bd4748  boost::function0<>::operator()()
> @  0x1ebf349  impala::Thread::SuperviseThread()
> @  0x1ec74e5  boost::_bi::list5<>::operator()<>()
> @  0x1ec7409  boost::_bi::bind_t<>::operator()()
> @  0x1ec73cc  boost::detail::thread_data<>::run()
> @  0x31a1f0a  thread_proxy
> @   0x36d1607851  (unknown)
> @   0x36d12e894d  (unknown)
> {code}
> Git hash of Impala used in job: ba84ad03cb83d7f7aed8524fcfbb0e2cdc9fdd53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7018) OpenSSL pending errors not cleared when spill-to-disk encryption is enabled

2018-05-15 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7018.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> OpenSSL pending errors not cleared when spill-to-disk encryption is enabled
> ---
>
> Key: IMPALA-7018
> URL: https://issues.apache.org/jira/browse/IMPALA-7018
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Hit DCHECK because of OpenSSL pending errors in my precommit job for enabling 
> disk spill encryption by default: 
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/212
> {noformat}
> F0509 19:35:46.537220 33290 openssl_util.h:201] Check failed: 
> ERR_peek_error() == 0 (101171331 vs. 0) Expected no pending OpenSSL errors on 
> kudu::Status 
> kudu::security::TlsContext::InitiateHandshake(kudu::security::TlsHandshakeType,
>  kudu::security::TlsHandshake*) const entry, but had: error:0607C083:digital 
> envelope routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 eF0509 19:35:46.537292 33290 openssl_util.h:201] Check 
> failed: ERR_peek_error() == 0 (101171331 vs. 0) Expected no pending OpenSSL 
> errors on kudu::Status 
> kudu::security::TlsContext::InitiateHandshake(kudu::security::TlsHandshakeType,
>  kudu::security::TlsHandshake*) const entry, but had: error:0607C083:digital 
> envelope routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610 
> error:0607C083:digital envelope routines:EVP_CIPHER_CTX_ctrl:no cipher 
> set:evp_enc.c:610 error:0607C083:digital envelope 
> routines:EVP_CIPHER_CTX_ctrl:no cipher set:evp_enc.c:610
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7025) PlannerTest.testTableSample has wrong mem-reservation

2018-05-16 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7025.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> PlannerTest.testTableSample has wrong mem-reservation
> -
>
> Key: IMPALA-7025
> URL: https://issues.apache.org/jira/browse/IMPALA-7025
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: broken-build, flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Seen once on master exhaustive:
> {noformat}
> Error Message
> Section PLAN of query:
> select id from functional_parquet.alltypes tablesample system(10) 
> repeatable(1234)
> Actual does not match expected result:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> |  Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB 
> thread-reservation=2
> ^^^
> PLAN-ROOT SINK
> |  mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=23.70KB
>stored statistics:
>  table: rows=unavailable size=unavailable
>  partitions: 0/24 rows=unavailable
>  columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=8.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable
> Expected:
> F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> |  Per-Host Resources: mem-estimate=16.00MB mem-reservation=16.00KB 
> thread-reservation=2
> PLAN-ROOT SINK
> |  mem-estimate=0B mem-reservation=0B thread-reservation=0
> |
> 00:SCAN HDFS [functional_parquet.alltypes]
>partitions=3/24 files=3 size=24.23KB
>stored statistics:
>  table: rows=unavailable size=unavailable
>  partitions: 0/24 rows=unavailable
>  columns: unavailable
>extrapolated-rows=disabled max-scan-range-rows=unavailable
>mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
>tuple-ids=0 row-size=4B cardinality=unavailable{noformat}
> This succeeded on the next build, so it is flaky and might not recur.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6645) Consider enabling disk spill encryption by default

2018-05-16 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6645.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Consider enabling disk spill encryption by default
> --
>
> Key: IMPALA-6645
> URL: https://issues.apache.org/jira/browse/IMPALA-6645
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Not Applicable
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> We currently have integrity checks disabled by default because of the 
> additional CPU cost. 
> Now that IMPALA-6219 is in, the integrity checks are cheaper so it may be 
> worth just paying the cost. We could also consider doing a lighter-weight CRC 
> checksum, but AES-GCM may be fast enough at 2.3GB/s to do by default.
> We should do an experiment to see if there's a major impact on spill-to-disk 
> perf.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6957) Include number of required threads in explain plan

2018-05-16 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6957.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Include number of required threads in explain plan
> --
>
> Key: IMPALA-6957
> URL: https://issues.apache.org/jira/browse/IMPALA-6957
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Frontend
>Affects Versions: Not Applicable
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Impala has an internal notion of "required threads" to execute a fragment, 
> e.g. the fragment execution thread and the first scanner thread. It's 
> possible to compute the number of required threads per fragment instance 
> based on the plan.
> We should include this in the resource profile and expose it in the explain 
> plan. This could then be a step toward implementing something like 
> IMPALA-6035.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7053) Reorganise query options into groups

2018-05-21 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7053:
-

 Summary: Reorganise query options into groups
 Key: IMPALA-7053
 URL: https://issues.apache.org/jira/browse/IMPALA-7053
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Reporter: Tim Armstrong
Assignee: Tim Armstrong


We have quite a lot of query options now and we're adding more for things like 
resource limits (e.g. IMPALA-6035). It's getting harder for users to understand 
the organisation and find relevant query options. We should consider grouping 
similar query options.

E.g. for this set of resource limits, we could reorganise in various ways:
* mem_limit -> resources.memory.per_node_limit
* buffer_pool_limit -> resources.memory.buffer_pool.per_node_limit
* thread_reservation_limit  -> resources.threads.per_node_limit
* thread_reservation_aggregate_limit -> resources.threads.aggregate_limit
* exec_time_limit_s -> resources.wallclock.limit_s

We could do the conversion incrementally. It would probably make sense to agree 
on a top-level organisation up-front.
* planner - anything that controls planner decisions like join ordering, etc
* scheduler - anything that controls scheduler decisions (admission control 
could maybe be included here too)
* resources - resource management functionality (limits, etc)
* session - anything related to session management like timeouts
* exec - anything that changes query execution behaviour (e.g. codegen, batch 
sizes, runtime filters, etc)
* Probably a group for anything that changes the semantic behaviour of a query 
(e.g. decimal_v2, appx_count_distinct, strict_mode, abort_on_error).
* A group that controls read and write behaviour of file formats like 
compression, etc



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7057) crash when DiskIoMgr::GetFreeBuffer(long*)+0xa9

2018-05-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7057.
---
Resolution: Duplicate

We changed the code to prevent the crash in Impala 2.7.0. I believe that the 
crash in that place most frequently required a corrupt file to trigger it (a 
negative page length would do it) - we tightened up the handling of that over 
several releases, notably with IMPALA-3917 in Impala 2.7.0 for Parquet

> crash when DiskIoMgr::GetFreeBuffer(long*)+0xa9
> ---
>
> Key: IMPALA-7057
> URL: https://issues.apache.org/jira/browse/IMPALA-7057
> Project: IMPALA
>  Issue Type: Bug
>Affects Versions: Impala 2.5.0
>Reporter: KarlManong
>Priority: Major
> Attachments: hs_err_pid1495848.log
>
>
>  impalad version 2.5.0-cdh5.7.1 RELEASE (build 
> 27a4325c18c2a01c7a8097681a0eccf6d4335ea1)。
> some query will cause impalad crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7058) RC and Seq fuzz tests cause crash

2018-05-23 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7058.
---
   Resolution: Fixed
 Assignee: Tim Armstrong  (was: Pranay Singh)
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

I disabled the test for now and created a follow-on JIRA to fix the remaining 
problems

> RC and Seq fuzz tests cause crash
> -
>
> Key: IMPALA-7058
> URL: https://issues.apache.org/jira/browse/IMPALA-7058
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Dimitris Tsirogiannis
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: broken-build, crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> The backtrace is here:
> {code:java}
> #7 0x02d89a84 in 
> impala::DelimitedTextParser::ParseFieldLocations (this=0xcf539a0, 
> max_tuples=1, remaining_len=-102, byte_buffer_ptr=0x7fc6b764dad0, 
> row_end_locations=0x7fc6b764dac0, field_locations=0x10034000, 
> num_tuples=0x7fc6b764dacc, num_fields=0x7fc6b764dac8, 
> next_column_start=0x7fc6b764dad8) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/delimited-text-parser.cc:205
> #8 0x01fdb641 in impala::HdfsSequenceScanner::ProcessRange 
> (this=0x15515f80, row_batch=0xcf54800) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-sequence-scanner.cc:352
> #9 0x02d7a20e in impala::BaseSequenceScanner::GetNextInternal 
> (this=0x15515f80, row_batch=0xcf54800) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/base-sequence-scanner.cc:181
> #10 0x01fb1ff0 in impala::HdfsScanner::ProcessSplit (this=0x15515f80) 
> at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scanner.cc:134
> #11 0x01f89258 in impala::HdfsScanNode::ProcessSplit 
> (this=0x2a4a8800, filter_ctxs=..., expr_results_pool=0x7fc6b764e4b0, 
> scan_range=0x13f5f8700, scanner_thread_reservation=0x7fc6b764e428) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:453
> #12 0x01f885f9 in impala::HdfsScanNode::ScannerThread 
> (this=0x2a4a8800, first_thread=false, scanner_thread_reservation=32768) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:360
> #13 0x01f87a6c in impala::HdfsScanNode::::operator()(void) 
> const (__closure=0x7fc6b764ebe8) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:292
> #14 0x01f89ac8 in 
> boost::detail::function::void_function_obj_invoker0,
>  void>::invoke(boost::detail::function::function_buffer &) 
> (function_obj_ptr=...) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #15 0x01bf0b28 in boost::function0::operator() 
> (this=0x7fc6b764ebe0) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #16 0x01edc57f in impala::Thread::SuperviseThread(std::string const&, 
> std::string const&, boost::function, impala::ThreadDebugInfo const*, 
> impala::Promise*) (name=..., category=..., functor=..., 
> parent_thread_info=0x7fc6b9e53890, thread_started=0x7fc6b9e527c0) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/repos/Impala/be/src/util/thread.cc:356
> #17 0x01ee471b in boost::_bi::list5 boost::_bi::value, boost::_bi::value, 
> boost::_bi::value, 
> boost::_bi::value >::operator() const&, std::string const&, boost::function, impala::ThreadDebugInfo 
> const*, impala::Promise*), boost::_bi::list0>(boost::_bi::type, 
> void (*&)(std::string const&, std::string const&, boost::function, 
> impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list0&, 
> int) (this=0x2a370fc0, f=@0x2a370fb8: 0x1edc218 
>  boost::function, impala::ThreadDebugInfo const*, 
> impala::Promise*)>, a=...) at 
> /data/jenkins/workspace/impala-asf-master-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #18 0x01ee463f in boost::_bi::bind_t const&, std::string const&, boost::function, impala::ThreadDebugInfo 
> const*, impala::Promise*), 
> boost::_bi::list5 boost::_bi::value,

[jira] [Created] (IMPALA-7065) Fix remaining invalid data handling bugs in RC and sequence file

2018-05-23 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7065:
-

 Summary: Fix remaining invalid data handling bugs in RC and 
sequence file
 Key: IMPALA-7065
 URL: https://issues.apache.org/jira/browse/IMPALA-7065
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: Tim Armstrong


IMPALA-7058 revealed some problems that weren't handled by the original 
IMPALA-3833 patch. We should fix the remaining cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7067) sleep(100000) command from test_shell_commandline.py can hang around and cause test_metrics_are_zero to fail

2018-05-23 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7067:
-

 Summary: sleep(10) command from test_shell_commandline.py can 
hang around and cause test_metrics_are_zero to fail
 Key: IMPALA-7067
 URL: https://issues.apache.org/jira/browse/IMPALA-7067
 Project: IMPALA
  Issue Type: Bug
  Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


{noformat}
03:25:47 [gw6] PASSED 
shell/test_shell_commandline.py::TestImpalaShell::test_cancellation 
...
03:27:01 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
03:27:01 verifier.verify_metrics_are_zero()
03:27:01 verifiers/metric_verifier.py:47: in verify_metrics_are_zero
03:27:01 self.wait_for_metric(metric, 0, timeout)
03:27:01 verifiers/metric_verifier.py:62: in wait_for_metric
03:27:01 self.impalad_service.wait_for_metric_value(metric_name, 
expected_value, timeout)
03:27:01 common/impala_service.py:135: in wait_for_metric_value
03:27:01 json.dumps(self.read_debug_webpage('rpcz?json')))
03:27:01 E   AssertionError: Metric value impala-server.mem-pool.total-bytes 
did not reach value 0 in 60s
{noformat}

I used the json dump from memz and the logs to trace it back to the 
sleep(10) query hanging around



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-5837) TestImpalaShell.test_cancellation fails on ASAN build: Could not execute command: select sleep(10000)

2018-05-22 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-5837.
---
Resolution: Cannot Reproduce

> TestImpalaShell.test_cancellation fails on ASAN build:  Could not execute 
> command: select sleep(1)
> --
>
> Key: IMPALA-5837
> URL: https://issues.apache.org/jira/browse/IMPALA-5837
> Project: IMPALA
>  Issue Type: Bug
>Affects Versions: Impala 2.10.0
>Reporter: David Knupp
>Priority: Minor
>  Labels: flaky
>
> This test has been marked as being flaky on the ASAN build in the past 
> (IMPALA-4046).
> This failure is different from that, so I'm not reopening that JIRA -- just 
> noting that it's the same test and the same build flavor.
> {noformat}
> 02:21:28 shell/test_shell_commandline.py:324: in test_cancellation
> 02:21:28 assert "Cancelling Query" in result.stderr, result.stderr
> 02:21:28 E   AssertionError: Starting Impala Shell without Kerberos 
> authentication
> 02:21:28 E Connected to localhost:21000
> 02:21:28 E Server version: impalad version 2.10.0-SNAPSHOT DEBUG (build 
> c67b198a19eda12c99905c2b9db30cc86f6e332d)
> 02:21:28 E Query: select sleep(1)
> 02:21:28 E Query submitted at: 2017-08-24 01:58:21 (Coordinator: 
> http://impala-boost-static-burst-slave-0228.vpc.cloudera.com:25000)
> 02:21:28 E Could not execute command: select sleep(1)
> 02:21:28 E 
> 02:21:28 E   assert 'Cancelling Query' in 'Starting Impala Shell without 
> Kerberos authentication\nConnected to localhost:21000\nServer version: 
> impalad version ... 
> http://impala-boost-static-burst-slave-0228.vpc.cloudera.com:25000)\nCould 
> not execute command: select sleep(1)\n'
> 02:21:28 E+  where 'Starting Impala Shell without Kerberos 
> authentication\nConnected to localhost:21000\nServer version: impalad version 
> ... 
> http://impala-boost-static-burst-slave-0228.vpc.cloudera.com:25000)\nCould 
> not execute command: select sleep(1)\n' = 
> .stderr
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7078) Selective Avro scans of wide tables use more memory than necessary

2018-05-25 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7078:
-

 Summary: Selective Avro scans of wide tables use more memory than 
necessary
 Key: IMPALA-7078
 URL: https://issues.apache.org/jira/browse/IMPALA-7078
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Affects Versions: Impala 2.12.0, Impala 2.11.0, Impala 2.10.0, Impala 2.13.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


We have some anecdotal evidence that this got worse somewhere between 2.7 and 
2.10, but it may be because of a change in timing rather than a direct 
consequence of a code change. The scan uses ~500MB of memory, but should 
probably be running with less.

Analysis revealed that most of the memory was from the tuple data buffers, 
which were multiple mb because they had space for 1024 wide rows.

There are a couple of things we could do:
* Directly improve Avro selective scans to avoid transferring excessive memory 
(i.e. something analogous to what IMPALA-4923 did for Parquet)
* Tweak the row batch queueing to reduce unnecessary queueing. If the consumer 
is running slow, lots of batches pile up in the queue. We saw the regression on 
a system with 24 disks, so the queue was ~280 entries, which is excessive.

Repro is as follows:
{code}
-- In Hive
create table wide_250_cols (
  col0 STRING, col1 STRING, col2 STRING, col3 STRING, col4 STRING, col5 STRING,
  col6 STRING, col7 STRING, col8 STRING, col9 STRING, col10 STRING, col11 
STRING,
  col12 STRING, col13 STRING, col1-- In Hive
create table wide_250_cols (
  col0 STRING, col1 STRING, col2 STRING, col3 STRING, col4 STRING, col5 STRING,
  col6 STRING, col7 STRING, col8 STRING, col9 STRING, col10 STRING, col11 
STRING,
  col12 STRING, col13 STRING, col14 STRING, col15 STRING, col16 STRING, col17 
STRING,
  col18 STRING, col19 STRING, col20 STRING, col21 STRING, col22 STRING, col23 
STRING,
  col24 STRING, col25 STRING, col26 STRING, col27 STRING, col28 STRING, col29 
STRING,
  col30 STRING, col31 STRING, col32 STRING, col33 STRING, col34 STRING, col35 
STRING,
  col36 STRING, col37 STRING, col38 STRING, col39 STRING, col40 STRING, col41 
STRING,
  col42 STRING, col43 STRING, col44 STRING, col45 STRING, col46 STRING, col47 
STRING,
  col48 STRING, col49 STRING, col50 STRING, col51 STRING, col52 STRING, col53 
STRING,
  col54 STRING, col55 STRING, col56 STRING, col57 STRING, col58 STRING, col59 
STRING,
  col60 STRING, col61 STRING, col62 STRING, col63 STRING, col64 STRING, col65 
STRING,
  col66 STRING, col67 STRING, col68 STRING, col69 STRING, col70 STRING, col71 
STRING,
  col72 STRING, col73 STRING, col74 STRING, col75 STRING, col76 STRING, col77 
STRING,
  col78 STRING, col79 STRING, col80 STRING, col81 STRING, col82 STRING, col83 
STRING,
  col84 STRING, col85 STRING, col86 STRING, col87 STRING, col88 STRING, col89 
STRING,
  col90 STRING, col91 STRING, col92 STRING, col93 STRING, col94 STRING, col95 
STRING,
  col96 STRING, col97 STRING, col98 STRING, col99 STRING, col100 STRING, col101 
STRING,
  col102 STRING, col103 STRING, col104 STRING, col105 STRING, col106 STRING, 
col107 STRING,
  col108 STRING, col109 STRING, col110 STRING, col111 STRING, col112 STRING, 
col113 STRING,
  col114 STRING, col115 STRING, col116 STRING, col117 STRING, col118 STRING, 
col119 STRING,
  col120 STRING, col121 STRING, col122 STRING, col123 STRING, col124 STRING, 
col125 STRING,
  col126 STRING, col127 STRING, col128 STRING, col129 STRING, col130 STRING, 
col131 STRING,
  col132 STRING, col133 STRING, col134 STRING, col135 STRING, col136 STRING, 
col137 STRING,
  col138 STRING, col139 STRING, col140 STRING, col141 STRING, col142 STRING, 
col143 STRING,
  col144 STRING, col145 STRING, col146 STRING, col147 STRING, col148 STRING, 
col149 STRING,
  col150 STRING, col151 STRING, col152 STRING, col153 STRING, col154 STRING, 
col155 STRING,
  col156 STRING, col157 STRING, col158 STRING, col159 STRING, col160 STRING, 
col161 STRING,
  col162 STRING, col163 STRING, col164 STRING, col165 STRING, col166 STRING, 
col167 STRING,
  col168 STRING, col169 STRING, col170 STRING, col171 STRING, col172 STRING, 
col173 STRING,
  col174 STRING, col175 STRING, col176 STRING, col177 STRING, col178 STRING, 
col179 STRING,
  col180 STRING, col181 STRING, col182 STRING, col183 STRING, col184 STRING, 
col185 STRING,
  col186 STRING, col187 STRING, col188 STRING, col189 STRING, col190 STRING, 
col191 STRING,
  col192 STRING, col193 STRING, col194 STRING, col195 STRING, col196 STRING, 
col197 STRING,
  col198 STRING, col199 STRING, col200 STRING, col201 STRING, col202 STRING, 
col203 STRING,
  col204 STRING, col205 STRING, col206 STRING, col207 STRING, col208 STRING, 
col209 STRING,
  col210 STRING, col211 STRING, col212 STRING, col213 STRING, col214 STRING, 
col215 STRING,
  col216 STRING, col217 STRING, col218 STRING, col219 STRING, col220 STRING, 
col221 STRING,
  col222 STRING, col223

[jira] [Created] (IMPALA-7069) Java UDF tests can trigger a crash in java.util.concurrent.ConcurrentHashMap.putVal()

2018-05-24 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7069:
-

 Summary: Java UDF tests can trigger a crash in 
java.util.concurrent.ConcurrentHashMap.putVal()
 Key: IMPALA-7069
 URL: https://issues.apache.org/jira/browse/IMPALA-7069
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Vuk Ercegovac
 Attachments: hs_err_pid22764.log, hs_err_pid29246.log, 
hs_err_pid8975.log, hs_err_pid9694.log

I hit this crash on a GVO, but was able to reproduce it on master on my desktop.

Repro steps:
{code}
git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
./buildall.sh -debug -noclean -notests -skiptests -ninja
start-impala-cluster.py
while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar' -n4 
--verbose; do date; done
{code}
I generally hit the crash within a hour of looping the test.

{noformat}
Stack: [0x7fa04791f000,0x7fa04812],  sp=0x7fa04811aff0,  free 
space=8175k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x8a8107]
V  [libjvm.so+0x96cf5f]
v  ~RuntimeStub::_complete_monitor_locking_Java
J 2758 C2 
java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object;
 (362 bytes) @ 0x7fa0c73637d4 [0x7fa0c7362d00+0xad4]
J 2311 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; 
(122 bytes) @ 0x7fa0c70a09a8 [0x7fa0c70a08e0+0xc8]
J 3953 C2 
java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; 
(40 bytes) @ 0x7fa0c71ce0f0 [0x7fa0c71ce0a0+0x50]
J 2987 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; 
(7 bytes) @ 0x7fa0c72ddb64 [0x7fa0c72ddb20+0x44]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x6648eb]
V  [libjvm.so+0x661ec4]
V  [libjvm.so+0x662523]
V  [libjvm.so+0x9e398d]
V  [libjvm.so+0x9e2326]
V  [libjvm.so+0x9e2b50]
V  [libjvm.so+0x42c099]
V  [libjvm.so+0x9dc786]
V  [libjvm.so+0x6a5edf]
V  [libjvm.so+0x6a70cb]  JVM_DefineClass+0xbb
V  [libjvm.so+0xa31ea5]
V  [libjvm.so+0xa37ea7]
J 4842  
sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class;
 (0 bytes) @ 0x7fa0c7af120b [0x7fa0c7af1100+0x10b]
J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5 
bytes) @ 0x7fa0c8cf2a74 [0x7fa0c8cf2940+0x134]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x6648eb]
V  [libjvm.so+0x6b5949]  JVM_DoPrivileged+0x429
J 1035  
java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
 (0 bytes) @ 0x7fa0c7220c7f [0x7fa0c7220bc0+0xbf]
J 20421 C2 
sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;
 (762 bytes) @ 0x7fa0c89bb848 [0x7fa0c89b9da0+0x1aa8]
J 4163 C2 
sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
 (104 bytes) @ 0x7fa0c789cca8 [0x7fa0c789c8c0+0x3e8]
J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes) @ 
0x7fa0c711c638 [0x7fa0c711c400+0x238]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x6648eb]
V  [libjvm.so+0x6822d7]
V  [libjvm.so+0x6862c9]
C  [impalad+0x2a004fa]  JNIEnv_::CallNonvirtualVoidMethodA(_jobject*, _jclass*, 
_jmethodID*, jvalue const*)+0x40
C  [impalad+0x29fe4ff]  
impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow 
const*) const+0x44b
C  [impalad+0x29ffde9]  
impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*, 
impala::TupleRow const*) const+0xbb
C  [impalad+0x2a0948a]  
impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, 
impala::TupleRow const*)+0x14c
C  [impalad+0x2a48eb1]  
impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*, 
impala::TupleRow const*) const+0x9d
C  [impalad+0x2a4abba]  impala_udf::BooleanVal 
impala::ScalarFnCall::InterpretEval(impala::ScalarExprEvaluator*,
 impala::TupleRow const*) const+0x18c
C  [impalad+0x2a4907d]  
impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*, 
impala::TupleRow const*) const+0x179
C  [impalad+0x2a09c7f]  
impala::ScalarExprEvaluator::GetBooleanVal(impala::TupleRow*)+0x37
C  [impalad+0x1b70efb]  
impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*, 
impala::TupleRow*)+0x23
C  [impalad+0x1b70efb]  
impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*, 
impala::TupleRow*)+0x23
C  [impalad+0x1b6fdf0]  
impala::ExecNode::EvalConjuncts(impala::ScalarExprEvaluator* const*, int, 
impala::TupleRow*)+0x42
C  [impalad+0x1bb60e3]  
impala::HdfsScanner::EvalConjuncts(impala::TupleRow*)+0x4d
C  [impalad+0x1bb08fd]  
impala::HdfsScanner::WriteCompleteTuple(impala::MemPool*,

[jira] [Resolved] (IMPALA-7055) test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to table format AVRO is not supported")

2018-05-24 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7055.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> test_avro_writer failing on upstream Jenkins (Expected exception: "Writing to 
> table format AVRO is not supported")
> --
>
> Key: IMPALA-7055
> URL: https://issues.apache.org/jira/browse/IMPALA-7055
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.0
>Reporter: David Knupp
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: correctness, flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> This failure occurred while verifying https://gerrit.cloudera.org/c/10455/, 
> but it is not related to that patch. The failing build is 
> https://jenkins.impala.io/job/gerrit-verify-dryrun/2511/ 
> (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2232/)
> Test appears to be (from 
> [avro-writer.test|https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/avro-writer.test]):
> {noformat}
>  QUERY
> SET ALLOW_UNSUPPORTED_FORMATS=0;
> insert into __avro_write select 1, "b", 2.2;
>  CATCH
> Writing to table format AVRO is not supported. Use query option 
> ALLOW_UNSUPPORTED_FORMATS
> {noformat}
> Error output:
> {noformat}
> 01:50:18 ] FAIL 
> query_test/test_compressed_formats.py::TestTableWriters::()::test_avro_writer[exec_option:
>  {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 01:50:18 ] === FAILURES 
> ===
> 01:50:18 ]  TestTableWriters.test_avro_writer[exec_option: {'batch_size': 0, 
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': 
> False, 'abort_on_error': 1, 'debug_action': None, 
> 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
> 01:50:18 ] [gw9] linux2 -- Python 2.7.12 
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 01:50:18 ] query_test/test_compressed_formats.py:189: in test_avro_writer
> 01:50:18 ] self.run_test_case('QueryTest/avro-writer', vector)
> 01:50:18 ] common/impala_test_suite.py:420: in run_test_case
> 01:50:18 ] assert False, "Expected exception: %s" % expected_str
> 01:50:18 ] E   AssertionError: Expected exception: Writing to table format 
> AVRO is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS
> 01:50:18 ]  Captured stderr setup 
> -
> 01:50:18 ] -- connecting to: localhost:21000
> 01:50:18 ] - Captured stderr call 
> -
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] use functional;
> 01:50:18 ] 
> 01:50:18 ] SET batch_size=0;
> 01:50:18 ] SET num_nodes=0;
> 01:50:18 ] SET disable_codegen_rows_threshold=5000;
> 01:50:18 ] SET disable_codegen=False;
> 01:50:18 ] SET abort_on_error=1;
> 01:50:18 ] SET exec_single_node_rows_threshold=0;
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] drop table if exists __avro_write;
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] 
> 01:50:18 ] create table __avro_write (i int, s string, d double)
> 01:50:18 ] stored as AVRO
> 01:50:18 ] TBLPROPERTIES ('avro.schema.literal'='{
> 01:50:18 ]   "name": "my_record",
> 01:50:18 ]   "type": "record",
> 01:50:18 ]   "fields": [
> 01:50:18 ]   {"name":"i", "type":["int", "null"]},
> 01:50:18 ]   {"name":"s", "type":["string", "null"]},
> 01:50:18 ]   {"name":"d", "type":["double", "null"]}]}');
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC=NONE;
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] 
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS=1;
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] 
> 01:50:18 ] insert into __avro_write select 0, "a", 1.1;
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET COMPRESSION_CODEC="";
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
> 01:50:18 ] SET ALLOW_UNSUPPORTED_FORMATS="0";
> 01:50:18 ] 
> 01:50:18 ] -- executing against localhost:21000
>

[jira] [Resolved] (IMPALA-7067) sleep(100000) command from test_shell_commandline.py can hang around and cause test_metrics_are_zero to fail

2018-05-24 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7067.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> sleep(10) command from test_shell_commandline.py can hang around and 
> cause test_metrics_are_zero to fail
> 
>
> Key: IMPALA-7067
> URL: https://issues.apache.org/jira/browse/IMPALA-7067
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: flaky
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {noformat}
> 03:25:47 [gw6] PASSED 
> shell/test_shell_commandline.py::TestImpalaShell::test_cancellation 
> ...
> 03:27:01 verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
> 03:27:01 verifier.verify_metrics_are_zero()
> 03:27:01 verifiers/metric_verifier.py:47: in verify_metrics_are_zero
> 03:27:01 self.wait_for_metric(metric, 0, timeout)
> 03:27:01 verifiers/metric_verifier.py:62: in wait_for_metric
> 03:27:01 self.impalad_service.wait_for_metric_value(metric_name, 
> expected_value, timeout)
> 03:27:01 common/impala_service.py:135: in wait_for_metric_value
> 03:27:01 json.dumps(self.read_debug_webpage('rpcz?json')))
> 03:27:01 E   AssertionError: Metric value impala-server.mem-pool.total-bytes 
> did not reach value 0 in 60s
> {noformat}
> I used the json dump from memz and the logs to trace it back to the 
> sleep(10) query hanging around



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6941) Allow loading more text scanner plugins

2018-05-18 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6941.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Allow loading more text scanner plugins
> ---
>
> Key: IMPALA-6941
> URL: https://issues.apache.org/jira/browse/IMPALA-6941
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> It would be nice if Impala supported loading plugins for scanning additional 
> text formats aside from LZO - the current logic is fairly specialized but 
> could easily be extended to load libraries for codecs like LZ4 and ZSTD if 
> available. It's kind of weird that we only support that one format.
> This might help a bit with IMPALA-6941 and IMPALA-3898 since we could test 
> the plugin-loading mechanism without relying on the external Impala-lzo 
> codebase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6995) False-positive DCHECK when converting whitespace-only strings to timestamp

2018-05-16 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6995.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> False-positive DCHECK when converting whitespace-only strings to timestamp
> --
>
> Key: IMPALA-6995
> URL: https://issues.apache.org/jira/browse/IMPALA-6995
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: crash
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {noformat}
> select cast(' ' as timestamp);
> {noformat}
> {noformat}
> F0508 14:32:07.245255 11824 timestamp-parse-util.cc:241] Check failed: 
> dt_ctx->fmt_len > 0 (0 vs. 0) 
> *** Check failure stack trace: ***
> @  0x428956d  google::LogMessage::Fail()
> @  0x428ae12  google::LogMessage::SendToLog()
> @  0x4288f47  google::LogMessage::Flush()
> @  0x428c50e  google::LogMessageFatal::~LogMessageFatal()
> @  0x1c3f485  impala::TimestampParser::ParseFormatTokensByStr()
> @  0x1c40553  impala::TimestampParser::Parse()
> @  0x1c4712a  impala::TimestampValue::Parse()
> @  0x2e5d8fa  impala::CastFunctions::CastToTimestampVal()
> @  0x2e45322  impala::ScalarFnCall::InterpretEval<>()
> @  0x2e27de5  impala::ScalarFnCall::GetTimestampVal()
> @  0x2de72de  impala::ScalarExprEvaluator::GetValue()
> @  0x2de6e69  impala::ScalarExprEvaluator::GetValue()
> @  0x1d1dbbf  
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow
> @ 0x7fb7cc1d07e8  (unknown)
> Picked up JAVA_TOOL_OPTIONS: 
> -agentlib:jdwp=transport=dt_socket,address=3,server=y,suspend=n 
> Wrote minidump to 
> /home/tarmstrong/Impala/incubator-impala/logs/cluster/minidumps/impalad/42afc7f9-5b4a-4ed7-b34ad782-d7904747.dmp
> {noformat}
> It seems to work fine on a release build.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6960) TestMtDopParquet.test_parquet_filtering flaky

2018-05-23 Thread Tim Armstrong (JIRA)


 [ 
https://issues.apache.org/jira/browse/IMPALA-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6960.
---
Resolution: Duplicate

I'll mark this as a dupe. If we see this specific failure happen again, I'll 
look at mitigating it.

> TestMtDopParquet.test_parquet_filtering flaky
> -
>
> Key: IMPALA-6960
> URL: https://issues.apache.org/jira/browse/IMPALA-6960
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: flaky
> Attachments: TEST-impala-parallel.log, TEST-impala-parallel.xml, 
> impalad.ip-172-31-21-122.ubuntu.log.INFO.20180502-174947.82370.gz, 
> impalad.ip-172-31-21-122.ubuntu.log.INFO.20180502-174948.82449, 
> impalad.ip-172-31-21-122.ubuntu.log.INFO.20180502-174949.82508
>
>
> {noformat}
> 19:28:35  TestMtDopParquet.test_parquet_filtering[mt_dop: 2 | exec_option: 
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
> 19:28:35 [gw0] linux2 -- Python 2.7.12 
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 19:28:35 query_test/test_mt_dop.py:103: in test_parquet_filtering
> 19:28:35 self.run_test_case('QueryTest/parquet-filtering', vector)
> 19:28:35 common/impala_test_suite.py:451: in run_test_case
> 19:28:35 verify_runtime_profile(test_section['RUNTIME_PROFILE'], 
> result.runtime_profile)
> 19:28:35 common/test_result_verifier.py:599: in verify_runtime_profile
> 19:28:35 "\n\nPROFILE:\n%s\n" % (function, field, expected_value, 
> actual_value, actual))
> 19:28:35 E   AssertionError: Aggregation of SUM over NumRowGroups did not 
> match expected results.
> 19:28:35 E   EXPECTED VALUE:
> 19:28:35 E   24
> 19:28:35 E   
> 19:28:35 E   ACTUAL VALUE:
> 19:28:35 E   21
> ... 
> 19:28:35 -- executing against localhost:21000
> 19:28:35 select count(*) from functional_parquet.alltypes where 
> date_string_col like '%/10';
> 19:28:35 
> 19:28:35 = 1 failed, 1899 passed, 60 skipped, 44 xfailed, 1 xpassed in 
> 2187.10 seconds ==
> {noformat}
> Query ID is be45f2c7cbf5354f:44e7abd3
> Could be the same problem as IMPALA-6338 with a missing profile. I'm going to 
> dig a little more to confirm that one of the fragment instance profiles i 
> missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7062) Fix unsafe RuntimeProfile::SortChildren() API

2018-05-23 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7062:
-

 Summary: Fix unsafe RuntimeProfile::SortChildren() API
 Key: IMPALA-7062
 URL: https://issues.apache.org/jira/browse/IMPALA-7062
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


As a follow-up to IMPALA-6484, we should fix the broken SortChildren() API that 
is not thread-safe and dangerous. We should replace it with something that 
sorts the children by total time using stable values that can't be modified 
concurrently by other threads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7078) Selective Avro scans of wide tables use more memory than necessary

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7078.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Selective Avro scans of wide tables use more memory than necessary
> --
>
> Key: IMPALA-7078
> URL: https://issues.apache.org/jira/browse/IMPALA-7078
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0, Impala 
> 2.13.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: avro, resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> We have some anecdotal evidence that this got worse somewhere between 2.7 and 
> 2.10, but it may be because of a change in timing rather than a direct 
> consequence of a code change. The scan uses ~500MB of memory, but should 
> probably be running with less.
> Analysis revealed that most of the memory was from the tuple data buffers, 
> which were multiple mb because they had space for 1024 wide rows.
> There are a couple of things we could do:
> * Directly improve Avro selective scans to avoid transferring excessive 
> memory (i.e. something analogous to what IMPALA-4923 did for Parquet)
> * Tweak the row batch queueing to reduce unnecessary queueing. If the 
> consumer is running slow, lots of batches pile up in the queue. We saw the 
> regression on a system with 24 disks, so the queue was ~280 entries, which is 
> excessive.
> Repro is as follows:
> {code}
> -- In Hive
> create table wide_250_cols (
>   col0 STRING, col1 STRING, col2 STRING, col3 STRING, col4 STRING, col5 
> STRING,
>   col6 STRING, col7 STRING, col8 STRING, col9 STRING, col10 STRING, col11 
> STRING,
>   col12 STRING, col13 STRING, col14 STRING, col15 STRING, col16 STRING, col17 
> STRING,
>   col18 STRING, col19 STRING, col20 STRING, col21 STRING, col22 STRING, col23 
> STRING,
>   col24 STRING, col25 STRING, col26 STRING, col27 STRING, col28 STRING, col29 
> STRING,
>   col30 STRING, col31 STRING, col32 STRING, col33 STRING, col34 STRING, col35 
> STRING,
>   col36 STRING, col37 STRING, col38 STRING, col39 STRING, col40 STRING, col41 
> STRING,
>   col42 STRING, col43 STRING, col44 STRING, col45 STRING, col46 STRING, col47 
> STRING,
>   col48 STRING, col49 STRING, col50 STRING, col51 STRING, col52 STRING, col53 
> STRING,
>   col54 STRING, col55 STRING, col56 STRING, col57 STRING, col58 STRING, col59 
> STRING,
>   col60 STRING, col61 STRING, col62 STRING, col63 STRING, col64 STRING, col65 
> STRING,
>   col66 STRING, col67 STRING, col68 STRING, col69 STRING, col70 STRING, col71 
> STRING,
>   col72 STRING, col73 STRING, col74 STRING, col75 STRING, col76 STRING, col77 
> STRING,
>   col78 STRING, col79 STRING, col80 STRING, col81 STRING, col82 STRING, col83 
> STRING,
>   col84 STRING, col85 STRING, col86 STRING, col87 STRING, col88 STRING, col89 
> STRING,
>   col90 STRING, col91 STRING, col92 STRING, col93 STRING, col94 STRING, col95 
> STRING,
>   col96 STRING, col97 STRING, col98 STRING, col99 STRING, col100 STRING, 
> col101 STRING,
>   col102 STRING, col103 STRING, col104 STRING, col105 STRING, col106 STRING, 
> col107 STRING,
>   col108 STRING, col109 STRING, col110 STRING, col111 STRING, col112 STRING, 
> col113 STRING,
>   col114 STRING, col115 STRING, col116 STRING, col117 STRING, col118 STRING, 
> col119 STRING,
>   col120 STRING, col121 STRING, col122 STRING, col123 STRING, col124 STRING, 
> col125 STRING,
>   col126 STRING, col127 STRING, col128 STRING, col129 STRING, col130 STRING, 
> col131 STRING,
>   col132 STRING, col133 STRING, col134 STRING, col135 STRING, col136 STRING, 
> col137 STRING,
>   col138 STRING, col139 STRING, col140 STRING, col141 STRING, col142 STRING, 
> col143 STRING,
>   col144 STRING, col145 STRING, col146 STRING, col147 STRING, col148 STRING, 
> col149 STRING,
>   col150 STRING, col151 STRING, col152 STRING, col153 STRING, col154 STRING, 
> col155 STRING,
>   col156 STRING, col157 STRING, col158 STRING, col159 STRING, col160 STRING, 
> col161 STRING,
>   col162 STRING, col163 STRING, col164 STRING, col165 STRING, col166 STRING, 
> col167 STRING,
>   col168 STRING, col169 STRING, col170 STRING, col171 STRING, col172 STRING, 
> col173 STRING,
>   col174 STRING, col175 STRING, col176 STRING, col177 STRING, col178 STRING, 
> col179 STRING,
>   col180 STRING, col181 STRING, col182 STRING, col183 STRING, col184 STRING, 
> col185 STRING,
>   col186 STRING, col187 STRING, col188 STRING, col189 STRING, col190 STRING, 
> col191 STRING,
>   col192 STRING, col193 STRING, col194 STRING, col195 STRING, col196 STRING, 
> col197 STRING,
>   col198 STRING, col199 STRING, col200 STRING, col201

[jira] [Resolved] (IMPALA-411) Add testing for ReadWriteUtil

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-411.
--
Resolution: Later

This isn't a terrible idea but we don't need a JIRA to track adding more tests.

> Add testing for ReadWriteUtil
> -
>
> Key: IMPALA-411
> URL: https://issues.apache.org/jira/browse/IMPALA-411
> Project: IMPALA
>  Issue Type: Task
>  Components: Backend
>Affects Versions: Impala 1.0
>Reporter: Nong Li
>Priority: Minor
>  Labels: ramp-up
>
> This file contains some low level byte utilities.  It is very unit testable 
> and we need a bit more coverage here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6199) Flaky test: metadata/test_hdfs_permissions.py

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6199.
---
Resolution: Cannot Reproduce

> Flaky test: metadata/test_hdfs_permissions.py
> -
>
> Key: IMPALA-6199
> URL: https://issues.apache.org/jira/browse/IMPALA-6199
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.11.0
>Reporter: Taras Bobrovytsky
>Priority: Critical
>
> TestHdfsPermissions.test_insert_into_read_only_table failed on a nightly 
> Isilon build with the following error message:
> {code}
>  TestHdfsPermissions.test_insert_into_read_only_table[exec_option: 
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 
> 'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
> [gw1] linux2 -- Python 2.6.6 
> /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/bin/../infra/python/env/bin/python
> metadata/test_hdfs_permissions.py:73: in test_insert_into_read_only_table
> self.hdfs_client.delete_file_dir('test-warehouse/%s' % TEST_TBL, 
> recursive=True)
> util/hdfs_util.py:90: in delete_file_dir
> if not self.exists(path):
> util/hdfs_util.py:138: in exists
> self.get_file_dir_status(path)
> util/hdfs_util.py:102: in get_file_dir_status
> return super(PyWebHdfsClientWithChmod, self).get_file_dir_status(path)
> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:338: in 
> get_file_dir_status
> _raise_pywebhdfs_exception(response.status_code, response.content)
> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:477: in 
> _raise_pywebhdfs_exception
> raise errors.PyWebHdfsException(msg=message)
> E   PyWebHdfsException: 
> E   
> E   403 Forbidden
> E   
> E   Forbidden
> E   You don't have permission to access /v1/html/500Text.html
> E   on this server.
> E   
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6953) Improve encapsulation within DiskIoMgr

2018-06-08 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6953.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Improve encapsulation within DiskIoMgr
> --
>
> Key: IMPALA-6953
> URL: https://issues.apache.org/jira/browse/IMPALA-6953
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> While DiskIoMgr is still fresh in my mind, I should do some refactoring to 
> improve the encapsulation within io::. Currently lots of classes are friends 
> with each other and some code is not in the most appropriate class.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7151) session-expiry-test failed - unable to open ThriftServer port

2018-06-08 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7151:
-

 Summary: session-expiry-test failed - unable to open ThriftServer 
port
 Key: IMPALA-7151
 URL: https://issues.apache.org/jira/browse/IMPALA-7151
 Project: IMPALA
  Issue Type: Bug
  Components: Infrastructure
Affects Versions: Impala 2.13.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
 Attachments: session-expiry-test.ERROR, session-expiry-test.FATAL, 
session-expiry-test.INFO, session-expiry-test.WARNING

{noformat}
ThriftServer 'backend' (on port: 52436) exited due to TException: Could not 
bind: Transport endpoint is not connected
{noformat}

This is similar to IMPALA-5499 but is on CentOS 7:
{noformat}
# lsb_release -a
LSB Version:
:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:CentOS Linux release 7.4.1708 (Core) 
Release:7.4.1708
Codename:   Core
{noformat}

It's in the ephemeral port range:
{noformat}
# cat /proc/sys/net/ipv4/ip_local_port_range
32768   60999
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-3359) expr-test: StartWithEphemeralPorts() crashes if impalad fails to start

2018-06-08 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-3359.
---
Resolution: Duplicate

> expr-test: StartWithEphemeralPorts() crashes if impalad fails to start
> --
>
> Key: IMPALA-3359
> URL: https://issues.apache.org/jira/browse/IMPALA-3359
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.5.0
>Reporter: Henry Robinson
>Assignee: Sailesh Mukil
>Priority: Minor
>
> If {{FindWithEphemeralPorts()}} returns the same port for several different 
> services in {{StartWithEphemeralPorts()}}, {{StartWithClientServers()}} will 
> crash when it calls {{delete impala}}.
> It might not be easy to fix the crash (as it's deep into Thrift), but maybe 
> let's make it so that {{StartWithEphemeralPorts()}} can't try to use the same 
> port for e.g. {{beeswax_port}} and {{hs2_port}}/
> h1. Log from {[expr-test}}
> {code}
> I0414 04:12:38.979552 16994 impala-server.cc:1837] Impala Beeswax Service 
> listening on 62874
> I0414 04:12:38.980504 16994 impala-server.cc:1859] Impala HiveServer2 Service 
> listening on 52011
> I0414 04:12:38.981437 16994 impala-server.cc:1879] ImpalaInternalService 
> listening on 52011
> I0414 04:12:38.981679 16994 thrift-server.cc:431] ThriftServer 'backend' 
> started on port: 52011 <- Duplicate ephemeral port
> E0414 04:12:38.983886 17166 thrift-server.cc:176] ThriftServer 
> 'hiveserver2-frontend' (on port: 52011) exited due to TException: Could not 
> bind: Transport endpoint is not connected
> E0414 04:12:38.984000 16994 thrift-server.cc:165] ThriftServer 
> 'hiveserver2-frontend' (on port: 52011) did not start correctly 
> {code}
> h1. Stacktrace
> {code}
> #0  0x74d6ecc9 in __GI_raise (sig=sig@entry=6) at 
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x74d720d8 in __GI_abort () at abort.c:89
> #2  0x74d67b86 in __assert_fail_base (fmt=0x74eb8830 "%s%s%s:%u: 
> %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x27d06b8 "!ret",
> file=file@entry=0x27d05c8 
> "/data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/thread/pthread/condition_variable_fwd.hpp",
>  line=line@entry=81,
> function=function@entry=0x27d2800 
>  
> "boost::condition_variable::~condition_variable()") at assert.c:92
> #3  0x74d67c32 in __GI___assert_fail (assertion=0x27d06b8 "!ret", 
> file=0x27d05c8 
> "/data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/thread/pthread/condition_variable_fwd.hpp",
>  line=81,
> function=0x27d2800 
>  
> "boost::condition_variable::~condition_variable()") at assert.c:101
> #4  0x01401b82 in boost::condition_variable::~condition_variable 
> (this=0x6f8ff20, __in_chrg=) at 
> /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/thread/pthread/condition_variable_fwd.hpp:81
> #5  0x0156f235 in impala::ImpalaServer::~ImpalaServer 
> (this=0x6f8fe00, __in_chrg=, __vtt_parm=) at 
> /data/henry/src/cloudera/Impala/be/src/service/impala-server.cc:706
> #6  0x0156f3c2 in impala::ImpalaServer::~ImpalaServer 
> (this=0x6f8fe00, __in_chrg=, __vtt_parm=) at 
> /data/henry/src/cloudera/Impala/be/src/service/impala-server.cc:706
> #7  0x015af1d2 in boost::checked_delete 
> (x=0x6f8fe00) at 
> /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
> #8  0x015bbf44 in 
> boost::detail::sp_counted_impl_p::dispose 
> (this=0x83bfa80) at 
> /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
> #9  0x0116226c in boost::detail::sp_counted_base::release 
> (this=0x83bfa80) at 
> /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
> #10 0x011622fb in boost::detail::shared_count::~shared_count 
> (this=0x83c9c00, __in_chrg=) at 
> /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/shared_count.hpp:443
> #11 0x01587ee2 in 
> boost::shared_ptr::~shared_ptr (this=0x83c9bf8, 
> __in_chrg=) at 
> /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/shared_ptr.hpp:323
> #12 0x015af6a4 in 
> impala::ImpalaInternalService::~ImpalaInternalService (this=0x83c9bf0, 
> __in_chrg=) at 
> /data/henry/src/cloudera/Impala/be/src/service/impala-internal-service.h:29
> #13 0x015af6e0 in 
> impala::ImpalaInternalService::~ImpalaInternalService (this=0x83c9bf0, 
> __in_chrg=) at 
> /data/henry/src/cloudera/Impala/be/src/service/impala-internal-service.h:29
> #14 0x015af718 in 
> boost::checked_delete (x=0x83c9bf0) at 
>

[jira] [Created] (IMPALA-7143) TestDescribeTableResults started failing because of OwnerType

2018-06-07 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7143:
-

 Summary: TestDescribeTableResults started failing because of 
OwnerType
 Key: IMPALA-7143
 URL: https://issues.apache.org/jira/browse/IMPALA-7143
 Project: IMPALA
  Issue Type: Bug
  Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Fredy Wijaya


{noformat}
23:27:32   
AuthorizationTest.TestDescribeTableResults:1963->verifyOutputWithOptionalData:1988
 Size difference. Expected: [# col_name, data_type, comment, , NULL, NULL, id, 
int, NULL, bool_col, boolean, NULL, tinyint_col, tinyint, NULL, smallint_col, 
smallint, NULL, int_col, int, NULL, bigint_col, bigint, NULL, float_col, float, 
NULL, double_col, double, NULL, date_string_col, string, NULL, string_col, 
string, NULL, timestamp_col, timestamp, NULL, , NULL, NULL, # Partition 
Information, NULL, NULL, # col_name, data_type, comment, , NULL, NULL, year, 
int, NULL, month, int, NULL, day, int, NULL, , NULL, NULL, # Detailed Table 
Information, NULL, NULL, Database:, functional, NULL, Owner:, *, NULL, 
CreateTime:, *, NULL, LastAccessTime:, UNKNOWN, NULL, Retention:, 0, NULL, 
Location:, hdfs://localhost:20500/test-warehouse/alltypesagg, NULL, Table 
Type:, EXTERNAL_TABLE, NULL, Table Parameters:, NULL, NULL, , 
DO_NOT_UPDATE_STATS, true, , EXTERNAL, TRUE, , STATS_GENERATED, TASK, , 
impala.lastComputeStatsTime, *, , numRows, 11000, , totalSize, 834279, , 
transient_lastDdlTime, *, , NULL, NULL, # Storage Information, NULL, NULL, 
SerDe Library:, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, NULL, 
InputFormat:, org.apache.hadoop.mapred.TextInputFormat, NULL, OutputFormat:, 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, NULL, Compressed:, 
No, NULL, Num Buckets:, 0, NULL, Bucket Columns:, [], NULL, Sort Columns:, [], 
NULL, Storage Desc Params:, NULL, NULL, , escape.delim, \\, , field.delim, ,, , 
serialization.format, ,] Actual: [# col_name, data_type, comment, , NULL, NULL, 
id, int, NULL, bool_col, boolean, NULL, tinyint_col, tinyint, NULL, 
smallint_col, smallint, NULL, int_col, int, NULL, bigint_col, bigint, NULL, 
float_col, float, NULL, double_col, double, NULL, date_string_col, string, 
NULL, string_col, string, NULL, timestamp_col, timestamp, NULL, , NULL, NULL, # 
Partition Information, NULL, NULL, # col_name, data_type, comment, , NULL, 
NULL, year, int, NULL, month, int, NULL, day, int, NULL, , NULL, NULL, # 
Detailed Table Information, NULL, NULL, Database:, functional, NULL, 
OwnerType:, USER, NULL, Owner:, jenkins, NULL, CreateTime:, Wed Jun 06 21:50:33 
PDT 2018, NULL, LastAccessTime:, UNKNOWN, NULL, Retention:, 0, NULL, Location:, 
hdfs://localhost:20500/test-warehouse/alltypesagg, NULL, Table Type:, 
EXTERNAL_TABLE, NULL, Table Parameters:, NULL, NULL, , DO_NOT_UPDATE_STATS, 
true, , EXTERNAL, TRUE, , STATS_GENERATED, TASK, , impala.lastComputeStatsTime, 
1528349431, , numRows, 11000, , totalSize, 834279, , transient_lastDdlTime, 
1528347033, , NULL, NULL, # Storage Information, NULL, NULL, SerDe Library:, 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, NULL, InputFormat:, 
org.apache.hadoop.mapred.TextInputFormat, NULL, OutputFormat:, 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, NULL, Compressed:, 
No, NULL, Num Buckets:, 0, NULL, Bucket Columns:, [], NULL, Sort Columns:, [], 
NULL, Storage Desc Params:, NULL, NULL, , escape.delim, \\, , field.delim, ,, , 
serialization.format, ,] expected:<150> but was:<153>
23:27:32   
AuthorizationTest.TestDescribeTableResults:1963->verifyOutputWithOptionalData:1988
 Size difference. Expected: [# col_name, data_type, comment, , NULL, NULL, id, 
int, NULL, bool_col, boolean, NULL, tinyint_col, tinyint, NULL, smallint_col, 
smallint, NULL, int_col, int, NULL, bigint_col, bigint, NULL, float_col, float, 
NULL, double_col, double, NULL, date_string_col, string, NULL, string_col, 
string, NULL, timestamp_col, timestamp, NULL, , NULL, NULL, # Partition 
Information, NULL, NULL, # col_name, data_type, comment, , NULL, NULL, year, 
int, NULL, month, int, NULL, day, int, NULL, , NULL, NULL, # Detailed Table 
Information, NULL, NULL, Database:, functional, NULL, Owner:, *, NULL, 
CreateTime:, *, NULL, LastAccessTime:, UNKNOWN, NULL, Retention:, 0, NULL, 
Location:, hdfs://localhost:20500/test-warehouse/alltypesagg, NULL, Table 
Type:, EXTERNAL_TABLE, NULL, Table Parameters:, NULL, NULL, , 
DO_NOT_UPDATE_STATS, true, , EXTERNAL, TRUE, , STATS_GENERATED, TASK, , 
impala.lastComputeStatsTime, *, , numRows, 11000, , totalSize, 834279, , 
transient_lastDdlTime, *, , NULL, NULL, # Storage Information, NULL, NULL, 
SerDe Library:, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, NULL, 
InputFormat:, org.apache.hadoop.mapred.TextInputFormat, NULL, OutputFormat:, 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, NULL, Compressed:, 
No, NULL, Num Buckets:,

[jira] [Created] (IMPALA-7145) Memory leak when looping test_spilling_naaj

2018-06-07 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7145:
-

 Summary: Memory leak when looping test_spilling_naaj
 Key: IMPALA-7145
 URL: https://issues.apache.org/jira/browse/IMPALA-7145
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


I tried looping this test for a while to flush out a different issue and 
discovered a memory leak:
{noformat}
while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose; 
do date; done
{noformat}

Eventually the test fails with "Memory Limit Exceeded" and there is a huge 
amount of untracked memory reported on /memz.

I tried running the heap growth profiler and the only suspicious thing I saw 
was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's 
something related to disk spill encryption and we haven't noticed it before 
because we had disk spill encryption off.

I'm looking the test with disk spill encryption disabled to see if the leak 
also occurs then.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7144) Reenable tests disabled by IMPALA-7143

2018-06-07 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7144:
-

 Summary: Reenable tests disabled by IMPALA-7143
 Key: IMPALA-7144
 URL: https://issues.apache.org/jira/browse/IMPALA-7144
 Project: IMPALA
  Issue Type: Bug
  Components: Frontend
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Fredy Wijaya


IMPALA-7143 should have unblocked tests but we need to get this test coverage 
back.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7143) TestDescribeTableResults started failing because of OwnerType

2018-06-07 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7143.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

Disabled the test for now, closing and opening a follow-on to reenable

> TestDescribeTableResults started failing because of OwnerType
> -
>
> Key: IMPALA-7143
> URL: https://issues.apache.org/jira/browse/IMPALA-7143
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Fredy Wijaya
>Priority: Blocker
>  Labels: broken-build
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> {noformat}
> 23:27:32   
> AuthorizationTest.TestDescribeTableResults:1963->verifyOutputWithOptionalData:1988
>  Size difference. Expected: [# col_name, data_type, comment, , NULL, NULL, 
> id, int, NULL, bool_col, boolean, NULL, tinyint_col, tinyint, NULL, 
> smallint_col, smallint, NULL, int_col, int, NULL, bigint_col, bigint, NULL, 
> float_col, float, NULL, double_col, double, NULL, date_string_col, string, 
> NULL, string_col, string, NULL, timestamp_col, timestamp, NULL, , NULL, NULL, 
> # Partition Information, NULL, NULL, # col_name, data_type, comment, , NULL, 
> NULL, year, int, NULL, month, int, NULL, day, int, NULL, , NULL, NULL, # 
> Detailed Table Information, NULL, NULL, Database:, functional, NULL, Owner:, 
> *, NULL, CreateTime:, *, NULL, LastAccessTime:, UNKNOWN, NULL, Retention:, 0, 
> NULL, Location:, hdfs://localhost:20500/test-warehouse/alltypesagg, NULL, 
> Table Type:, EXTERNAL_TABLE, NULL, Table Parameters:, NULL, NULL, , 
> DO_NOT_UPDATE_STATS, true, , EXTERNAL, TRUE, , STATS_GENERATED, TASK, , 
> impala.lastComputeStatsTime, *, , numRows, 11000, , totalSize, 834279, , 
> transient_lastDdlTime, *, , NULL, NULL, # Storage Information, NULL, NULL, 
> SerDe Library:, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, NULL, 
> InputFormat:, org.apache.hadoop.mapred.TextInputFormat, NULL, OutputFormat:, 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, NULL, 
> Compressed:, No, NULL, Num Buckets:, 0, NULL, Bucket Columns:, [], NULL, Sort 
> Columns:, [], NULL, Storage Desc Params:, NULL, NULL, , escape.delim, \\, , 
> field.delim, ,, , serialization.format, ,] Actual: [# col_name, data_type, 
> comment, , NULL, NULL, id, int, NULL, bool_col, boolean, NULL, tinyint_col, 
> tinyint, NULL, smallint_col, smallint, NULL, int_col, int, NULL, bigint_col, 
> bigint, NULL, float_col, float, NULL, double_col, double, NULL, 
> date_string_col, string, NULL, string_col, string, NULL, timestamp_col, 
> timestamp, NULL, , NULL, NULL, # Partition Information, NULL, NULL, # 
> col_name, data_type, comment, , NULL, NULL, year, int, NULL, month, int, 
> NULL, day, int, NULL, , NULL, NULL, # Detailed Table Information, NULL, NULL, 
> Database:, functional, NULL, OwnerType:, USER, NULL, Owner:, jenkins, NULL, 
> CreateTime:, Wed Jun 06 21:50:33 PDT 2018, NULL, LastAccessTime:, UNKNOWN, 
> NULL, Retention:, 0, NULL, Location:, 
> hdfs://localhost:20500/test-warehouse/alltypesagg, NULL, Table Type:, 
> EXTERNAL_TABLE, NULL, Table Parameters:, NULL, NULL, , DO_NOT_UPDATE_STATS, 
> true, , EXTERNAL, TRUE, , STATS_GENERATED, TASK, , 
> impala.lastComputeStatsTime, 1528349431, , numRows, 11000, , totalSize, 
> 834279, , transient_lastDdlTime, 1528347033, , NULL, NULL, # Storage 
> Information, NULL, NULL, SerDe Library:, 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, NULL, InputFormat:, 
> org.apache.hadoop.mapred.TextInputFormat, NULL, OutputFormat:, 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, NULL, 
> Compressed:, No, NULL, Num Buckets:, 0, NULL, Bucket Columns:, [], NULL, Sort 
> Columns:, [], NULL, Storage Desc Params:, NULL, NULL, , escape.delim, \\, , 
> field.delim, ,, , serialization.format, ,] expected:<150> but was:<153>
> 23:27:32   
> AuthorizationTest.TestDescribeTableResults:1963->verifyOutputWithOptionalData:1988
>  Size difference. Expected: [# col_name, data_type, comment, , NULL, NULL, 
> id, int, NULL, bool_col, boolean, NULL, tinyint_col, tinyint, NULL, 
> smallint_col, smallint, NULL, int_col, int, NULL, bigint_col, bigint, NULL, 
> float_col, float, NULL, double_col, double, NULL, date_string_col, string, 
> NULL, string_col, string, NULL, timestamp_col, timestamp, NULL, , NULL, NULL, 
> # Partition Information, NULL, NULL, # col_name, data_type, comment, , NULL, 
> NULL, year, int, NULL, month, int, NULL, day, int, NULL, , NULL, NULL, # 
> Detailed Table Information, NULL, NULL, Database:, functional, NULL, Owner:, 
> *, NULL, CreateTime:, *, NULL, LastAccessTime:, UNKNOWN, NULL, Retention:, 0, 
> NULL, Location:,

[jira] [Created] (IMPALA-7146) log session ID in Impala daemon log

2018-06-07 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7146:
-

 Summary: log session ID in Impala daemon log
 Key: IMPALA-7146
 URL: https://issues.apache.org/jira/browse/IMPALA-7146
 Project: IMPALA
  Issue Type: Improvement
  Components: Distributed Exec
Reporter: Tim Armstrong
Assignee: Tim Armstrong


We should log the session_id to query_id mapping to enable debugging

This would be very useful for investigating issues with idle_session_timeout, 
etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7126) test_spilling_naaj failed: "verification of read data failed"

2018-06-07 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7126.
---
Resolution: Cannot Reproduce

> test_spilling_naaj failed: "verification of read data failed"
> -
>
> Key: IMPALA-7126
> URL: https://issues.apache.org/jira/browse/IMPALA-7126
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
>  Labels: flaky
>
> {noformat}
> 14:07:04 === FAILURES 
> ===
> 14:07:04  TestSpillingDebugActionDimensions.test_spilling_naaj[exec_option: 
> {'debug_action': None, 'default_spillable_buffer_size': '256k'} | 
> table_format: parquet/none] 
> 14:07:04 [gw0] linux2 -- Python 2.6.6 
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/bin/../infra/python/env/bin/python
> 14:07:04 query_test/test_spilling.py:71: in test_spilling_naaj
> 14:07:04 self.run_test_case('QueryTest/spilling-naaj', vector)
> 14:07:04 common/impala_test_suite.py:405: in run_test_case
> 14:07:04 result = self.__execute_query(target_impalad_client, query, 
> user=user)
> 14:07:04 common/impala_test_suite.py:620: in __execute_query
> 14:07:04 return impalad_client.execute(query, user=user)
> 14:07:04 common/impala_connection.py:160: in execute
> 14:07:04 return self.__beeswax_client.execute(sql_stmt, user=user)
> 14:07:04 beeswax/impala_beeswax.py:173: in execute
> 14:07:04 handle = self.__execute_query(query_string.strip(), user=user)
> 14:07:04 beeswax/impala_beeswax.py:341: in __execute_query
> 14:07:04 self.wait_for_completion(handle)
> 14:07:04 beeswax/impala_beeswax.py:361: in wait_for_completion
> 14:07:04 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> 14:07:04 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
> 14:07:04 EQuery aborted:Error reading 65536 bytes from scratch file 
> '/tmp/impala-scratch/d543584c89768b84:ec655168_9b33ae1f-6775-4acf-93ee-5823de190d2c'
>  on backend ec2-m2-4xlarge-centos-6-4-1fae.vpc.cloudera.com:22002 at offset 
> 72155136: verification of read data failed.
> {noformat}
> Could either be a software bug or a disk error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7147) Hit DCHECK in Parquet fuzz test

2018-06-07 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7147:
-

 Summary: Hit DCHECK in Parquet fuzz test
 Key: IMPALA-7147
 URL: https://issues.apache.org/jira/browse/IMPALA-7147
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Tim Armstrong
Assignee: Csaba Ringhofer


This test was running:

{noformat}
10:24:44 [gw3] ERROR 
query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option:
 {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5', 
'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: 
parquet/none] 
{noformat}

{noformat}
F0607 10:23:52.436451  9895 bit-stream-utils.inline.h:111] Check failed: 
bit_width <= sizeof(T) * 8 (9 vs. 8) 
{noformat}
{noformat}
CORE: ./core.1528392237.27349.impalad
BINARY: ./be/build/latest/service/impalad
Core was generated by 
`/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/build/late'.
Program terminated with signal 6, Aborted.
#0  0x7f30633008e5 in raise () from /lib64/libc.so.6
To enable execution of this file add
add-auto-load-safe-path 
/data/jenkins/workspace/impala-asf-master-core-local/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
line to your configuration file "/var/lib/jenkins/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/var/lib/jenkins/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
info "(gdb)Auto-loading safe path"
#0  0x7f30633008e5 in raise () from /lib64/libc.so.6
#1  0x7f30633020c5 in abort () from /lib64/libc.so.6
#2  0x042bdf64 in google::DumpStackTraceAndExit() ()
#3  0x042b49dd in google::LogMessage::Fail() ()
#4  0x042b6282 in google::LogMessage::SendToLog() ()
#5  0x042b43b7 in google::LogMessage::Flush() ()
#6  0x042b797e in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x022d001f in impala::BatchedBitReader::UnpackAndDecodeBatch (this=0xe8cc720, bit_width=9, dict=0xef5a720 "\a\b\t", dict_len=10, 
num_values=128, v=0xe8cc800 "") at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/bit-stream-utils.inline.h:111
#8  0x022ccc56 in impala::RleBatchDecoder::DecodeLiteralValues (this=0xe8cc720, 
num_literals_to_consume=128, dict=0xef5a720 "\a\b\t", dict_len=10, 
values=0xe8cc800 "") at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/rle-encoding.h:569
#9  0x022ca1a8 in impala::DictDecoder::DecodeNextValue 
(this=0xe8cc718, value=0x1cd00c057 "") at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/dict-encoding.h:469
#10 0x022a8c3f in GetNextValue (this=0xe8cc480, max_values=1, 
tuple_size=90, tuple_mem=0x1cd00c000 "", num_values=0x7f2f9e5ad600) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/util/dict-encoding.h:447
#11 DecodeValue<(parquet::Encoding::type)2> (this=0xe8cc480, max_values=1, 
tuple_size=90, tuple_mem=0x1cd00c000 "", num_values=0x7f2f9e5ad600) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:625
#12 ReadSlot<(parquet::Encoding::type)2, false> (this=0xe8cc480, 
max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "", 
num_values=0x7f2f9e5ad600) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:606
#13 impala::ScalarColumnReader::MaterializeValueBatch 
(this=0xe8cc480, max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "", 
num_values=0x7f2f9e5ad600) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:555
#14 0x022914df in impala::ScalarColumnReader::MaterializeValueBatch (this=0xe8cc480, 
max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "", 
num_values=0x7f2f9e5ad600) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:581
#15 0x02278854 in impala::ScalarColumnReader::ReadValueBatch (this=0xe8cc480, 
max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "", 
num_values=0xab914b0) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:513
#16 0x022627b1 in impala::ScalarColumnReader::ReadNonRepeatedValueBatch (this=0xe8cc480, 
pool=0xab914f8, max_values=1, tuple_size=90, tuple_mem=0x1cd00c000 "", 
num_values=0xab914b0) at 
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/be/src/exec/parquet-column-readers.cc:228
#17 0x02095768 in impala::HdfsParquetScanner::AssembleRows 
(this=0x1ba692c00, column_readers=..., row_batch=0x1ba5bd040,

[jira] [Created] (IMPALA-7156) Many authorization tests failed with "User 'jenkins' does not have privileges to access"

2018-06-08 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7156:
-

 Summary: Many authorization tests failed with "User 'jenkins' does 
not have privileges to access"
 Key: IMPALA-7156
 URL: https://issues.apache.org/jira/browse/IMPALA-7156
 Project: IMPALA
  Issue Type: Bug
  Components: Infrastructure
Affects Versions: Impala 2.13.0
Reporter: Tim Armstrong
Assignee: Fredy Wijaya


Failed tests:
{noformat}

authorization.test_grant_revoke.TestGrantRevoke.test_role_privilege_case[exec_option:
 {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None, 
'exec_single_node_rows_threshold': 0} | table_format: text/none]
org.apache.impala.analysis.AuthorizationTest.TestServerLevelInsert[1]
org.apache.impala.analysis.AuthorizationTest.TestFunction[1]
org.apache.impala.analysis.AuthorizationTest.TestDropDatabase[1]
org.apache.impala.analysis.AuthorizationTest.TestServerLevelSelect[1]
org.apache.impala.analysis.AuthorizationTest.TestTruncate[1]
org.apache.impala.analysis.AuthorizationTest.TestShowCreateTable[1]
org.apache.impala.analysis.AuthorizationTest.TestShortUsernameUsed[1]
org.apache.impala.analysis.AuthorizationTest.TestComputeStatsTable[1]
org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns[1]
org.apache.impala.analysis.AuthorizationTest.TestLoad[1]
org.apache.impala.analysis.AuthorizationTest.TestCreateDatabase[1]
org.apache.impala.analysis.AuthorizationTest.TestInsert[1]
org.apache.impala.analysis.AuthorizationTest.TestSelect[1]
org.apache.impala.analysis.AuthorizationTest.TestCreateView[1]
org.apache.impala.analysis.AuthorizationTest.TestServerLevelAlter[1]
org.apache.impala.analysis.AuthorizationTest.TestResetMetadata[1]
org.apache.impala.analysis.AuthorizationTest.TestExplain[1]
org.apache.impala.analysis.AuthorizationTest.TestCommentOn[1]
org.apache.impala.analysis.AuthorizationTest.TestWithClause[1]
org.apache.impala.analysis.AuthorizationTest.TestAlterView[1]
org.apache.impala.analysis.AuthorizationTest.TestShowPermissions[1]
org.apache.impala.analysis.AuthorizationTest.TestServerLevelDrop[1]
org.apache.impala.analysis.AuthorizationTest.TestDropView[1]
org.apache.impala.analysis.AuthorizationTest.TestShowDbResultsFiltered[1]
org.apache.impala.analysis.AuthorizationTest.TestShowTableResultsFiltered[1]
org.apache.impala.analysis.AuthorizationTest.TestUnion[1]
org.apache.impala.analysis.AuthorizationTest.TestUseDb[1]
org.apache.impala.analysis.AuthorizationTest.TestDropStats[1]
org.apache.impala.analysis.AuthorizationTest.TestDropTable[1]
org.apache.impala.analysis.AuthorizationTest.TestHs2GetSchema[1]
org.apache.impala.analysis.AuthorizationTest.TestHs2GetTables[1]
org.apache.impala.analysis.AuthorizationTest.TestDescribeDb[1]
org.apache.impala.analysis.AuthorizationTest.TestAlterTable[1]
org.apache.impala.analysis.AuthorizationTest.TestDescribe[1]
org.apache.impala.analysis.AuthorizationTest.TestServerLevelCreate[1]
org.apache.impala.analysis.AuthorizationTest.TestCreateTable[1]
{noformat}

The errors are all along the lines of:
{noformat}
org.apache.impala.catalog.AuthorizationException: User 'jenkins' does not have 
privileges to execute 'INSERT' on: functional_parquet.alltypes
at 
org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:146)
at 
org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:567)
at 
org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:530)
at 
org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:409)
at 
org.apache.impala.common.FrontendTestBase.parseAndAnalyze(FrontendTestBase.java:429)
at 
org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2841)
at 
org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2836)
at 
org.apache.impala.analysis.AuthorizationTest.AuthzOk(AuthorizationTest.java:2832)
at 
org.apache.impala.analysis.AuthorizationTest.TestTruncate(AuthorizationTest.java:1417)

{noformat}

{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.impala.analysis.AuthorizationTest.TestHs2GetColumns(AuthorizationTest.java:2140)
{noformat}
Seen on commit b8f3d3cc6d8c3aa16e3004433a2f54a94b27a8cc



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7185) Reduce statestore frequency for custom cluster tests by default

2018-06-18 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7185:
-

 Summary: Reduce statestore frequency for custom cluster tests by 
default
 Key: IMPALA-7185
 URL: https://issues.apache.org/jira/browse/IMPALA-7185
 Project: IMPALA
  Issue Type: Bug
  Components: Infrastructure
Reporter: Tim Armstrong
Assignee: Tim Armstrong


It takes several seconds to run the first query after cluster startup because 
of the statestore propagation delay for the catalog, which adds some real time 
to custom cluster tests. We should think about lowering the default update 
interval for those tests to make them start up faster.

We could just prefix the statestored_args with lower values, allowing 
individual tests to override if needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7145) Leak of memory from OpenSSL when spill-to-disk encryption is enabled

2018-06-12 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7145.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Leak of memory from OpenSSL when spill-to-disk encryption is enabled
> 
>
> Key: IMPALA-7145
> URL: https://issues.apache.org/jira/browse/IMPALA-7145
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, 
> Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 
> 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, 2.2.0, 2.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> I tried looping this test for a while to flush out a different issue and 
> discovered a memory leak:
> {noformat}
> while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose; 
> do date; done
> {noformat}
> Eventually the test fails with "Memory Limit Exceeded" and there is a huge 
> amount of untracked memory reported on /memz.
> I tried running the heap growth profiler and the only suspicious thing I saw 
> was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's 
> something related to disk spill encryption and we haven't noticed it before 
> because we had disk spill encryption off.
> If I disable encryption "start-impala-cluster.py 
> --impalad_args=--disk_spill_encryption=false" then I don't see the leak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-7164) Define public API for RuntimeProfile

2018-06-12 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7164:
-

 Summary: Define public API for RuntimeProfile
 Key: IMPALA-7164
 URL: https://issues.apache.org/jira/browse/IMPALA-7164
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Reporter: Tim Armstrong


Currently the only public API for the runtime profile is the Thrift definition. 
The exact layout and counters are all implementation-dependent and subject to 
change. People are trying to build tools that consume runtime profiles and 
process them, but that's hard since the counters can change from version to 
version and sometimes the semantics change.

I think we need to figure out which things are part of the public API, validate 
that they make sense and document them clearly. We should also clearly document 
that things outside of this public API are subject to change without notice. I 
don't think the public API necessarily needs to be "porcelain", but we should 
generally try to avoid unnecessary changes and mention any changes in release 
notes etc.

We could start simple and just collect "public" counter names in a module and 
comment each of them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-5216) Admission control queuing should be asynchronous

2018-06-13 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-5216.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

Resolving on Bikram's behalf - he's afk for a bit.

> Admission control queuing should be asynchronous
> 
>
> Key: IMPALA-5216
> URL: https://issues.apache.org/jira/browse/IMPALA-5216
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Backend
>Reporter: Dan Hecht
>Assignee: Bikramjeet Vig
>Priority: Major
>  Labels: admission-control, resource-management
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Currently, admission control queuing occurs synchronously w.r.t. the 
> {{ExecuteStatement}} client RPC. That is, a query handle is not returned 
> until the query is admitted.
> Instead, the queuing should occur on the asynchronous path.  That way, the 
> client gets a query handle back immediately and can e.g. cancel a query that 
> is in the admission control queue.
> We'll also need a way to better expose the progress of a query handle 
> (related to IMPALA-124). E.g. that the query is queued for admission and what 
> resource(s) it's waiting on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7174) TestAdmissionController.test_cancellation failed with incorrect total-admitted metric

2018-06-15 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7174.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> TestAdmissionController.test_cancellation failed with incorrect 
> total-admitted metric
> -
>
> Key: IMPALA-7174
> URL: https://issues.apache.org/jira/browse/IMPALA-7174
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: broken-build
> Fix For: Impala 2.13.0, Impala 3.1.0
>
> Attachments: 
> impalad.ec2-m2-4xlarge-centos-6-4-01f7.vpc.cloudera.com.jenkins.log.INFO.20180614-060607.2553
>
>
> The failed revision is  ee9a9b6c5000cf915716a15ea8a0b3605290a9a5, an 
> descendant of  'IMPALA-5216: Make admission control queuing async'.
> {noformat}
> Stacktrace
> custom_cluster/test_admission_controller.py:557: in test_cancellation
> assert self.cluster.impalads[0].service.get_metric_value(
> E   assert 0 == 3
> E+  where 0 =   0x4ed1b90>>('admission-controller.total-admitted.default-pool')
> E+where  > = 
>  0x4ed1b90>.get_metric_value
> E+  where  0x4ed1b90> =  0x52474d0>.service
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7111) ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled

2018-06-15 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7111.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

commit 76d40f92ad9214053602518adaf00df8b7477a37
Author: Tim Armstrong 
Date: Wed Jun 13 13:48:48 2018 -0700

IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
 
 This is an attempt to either avoid the bug or make it easier to diagnose
 if it reoccurs. My suspicion is that somehow boost::split() is
 accessing the input string in a non-thread-safe manner, but the
 implementation is opaque enough that it's not obvious how.
 
 Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
 Reviewed-on: http://gerrit.cloudera.org:8080/10709
 Reviewed-by: Impala Public Jenkins 
 Tested-by: Impala Public Jenkins 

> ASAN heap-use-after-free in impala::HdfsPluginTextScanner::CheckPluginEnabled
> -
>
> Key: IMPALA-7111
> URL: https://issues.apache.org/jira/browse/IMPALA-7111
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Lars Volker
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: asan, broken-build
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
>  [~tarmstr...@cloudera.com] - I'm assigning this to you since you added this 
> file in IMPALA-6941.
> {noformat}
> ==4582==ERROR: AddressSanitizer: heap-use-after-free on address 
> 0x603000e8aa28 at pc 0x017ab9b4 bp 0x7f67e5f6b650 sp 0x7f67e5f6b648
> READ of size 1 at 0x603000e8aa28 thread T9236
> #0 0x17ab9b3 in bool 
> __gnu_cxx::__ops::_Iter_pred 
> >::operator()<__gnu_cxx::__normal_iterator 
> >(__gnu_cxx::__normal_iterator) 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/predefined_ops.h:231:24
> #1 0x17ab745 in __gnu_cxx::__normal_iterator 
> std::__find_if<__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__ops::_Iter_pred > 
> >(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator, 
> __gnu_cxx::__ops::_Iter_pred >, 
> std::random_access_iterator_tag) 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:140:8
> #2 0x17ab2dc in __gnu_cxx::__normal_iterator 
> std::__find_if<__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__ops::_Iter_pred > 
> >(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator, 
> __gnu_cxx::__ops::_Iter_pred >) 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:161:14
> #3 0x17aaf6c in __gnu_cxx::__normal_iterator 
> std::find_if<__gnu_cxx::__normal_iterator, 
> boost::algorithm::detail::is_any_ofF 
> >(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator, 
> boost::algorithm::detail::is_any_ofF) 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/bits/stl_algo.h:3803:14
> #4 0x17aaba1 in boost::iterator_range<__gnu_cxx::__normal_iterator std::string> > 
> boost::algorithm::detail::token_finderF
>  >::operator()<__gnu_cxx::__normal_iterator 
> >(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator) const 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/finder.hpp:565:41
> #5 0x17ac118 in 
> boost::function2 std::string> >, __gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator 
> >::operator()(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator) const 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #6 0x17abf8d in 
> boost::algorithm::detail::find_iterator_base<__gnu_cxx::__normal_iterator  std::string> >::do_find(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator) const 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/detail/find_iterator.hpp:63:32
> #7 0x17aa00c in 
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string> >::increment() 
> /data/jenkins/workspace/impala-asf-2.x-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/algorithm/string/find_iterator.hpp:305:44
> #8 0x17a95a5 in 
> boost::algorithm::split_iterator<__gnu_cxx::__normal_iterator std::string> 
> >::split_iterator
>  > >(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator, 
> boost::algorithm::detail::token_finderF
>  >) 
>

[jira] [Created] (IMPALA-7179) Consider changing --allow_multiple_scratch_dirs_per_device default

2018-06-15 Thread Tim Armstrong (JIRA)

Tim Armstrong created IMPALA-7179:
-

 Summary: Consider changing 
--allow_multiple_scratch_dirs_per_device default
 Key: IMPALA-7179
 URL: https://issues.apache.org/jira/browse/IMPALA-7179
 Project: IMPALA
  Issue Type: Sub-task
  Components: Backend
Reporter: Tim Armstrong
Assignee: Tim Armstrong


I've seen multiple instances of Impala users being tripped up by this behaviour 
and zero instances of it being useful (although it's possible that it helped 
someone and they didn't notice). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6035) Add query option that rejects queries based on query complexity

2018-06-15 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-6035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6035.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Add query option that rejects queries based on query complexity
> ---
>
> Key: IMPALA-6035
> URL: https://issues.apache.org/jira/browse/IMPALA-6035
> Project: IMPALA
>  Issue Type: Sub-task
>  Components: Distributed Exec
>Reporter: Mostafa Mokhtar
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: admission-control
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> Expose query option to reject queries with a large number of remote 
> fragments. 
> Queries with a large number of fragments can negatively affect cluster 
> behavior. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-5604) Document DISABLE_CODEGEN_ROWS_THRESHOLD

2018-06-15 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-5604.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

commit 1b0a027ffb216d729fe68e7240313af77550ae08
Author: Tim Armstrong 
Date: Thu Jun 14 16:13:58 2018 -0700

IMPALA-5604: document DISABLE_CODEGEN_ROWS_THRESHOLD
 
 Also fix a couple of nits in EXEC_SINGLE_NODE_ROWS_THRESHOLD.
 
 Change-Id: I709cd55e3869888feb645f85e61a99901d41d479
 Reviewed-on: http://gerrit.cloudera.org:8080/10727
 Reviewed-by: Alex Rodoni 
 Tested-by: Impala Public Jenkins 

> Document DISABLE_CODEGEN_ROWS_THRESHOLD
> ---
>
> Key: IMPALA-5604
> URL: https://issues.apache.org/jira/browse/IMPALA-5604
> Project: IMPALA
>  Issue Type: Task
>  Components: Docs
>Affects Versions: Impala 2.10.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> We just checking in DISABLE_CODEGEN_ROWS_THRESHOLD to master with default 
> value 50,000.
> The option uses the planner's estimates to determine whether to disable 
> codegen for a query based on the maximum number of rows flowing through any 
> part of the query plan. 
> It is similar to EXEC_SINGLE_NODE_ROWS_THRESHOLD in behaviour except the 
> threshold is per-node rows instead of total rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-646) Handle gracefully an LZO-text table containing regular text files

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-646.
--
Resolution: Cannot Reproduce

We're a lot more lenient here, especially after IMPALA-6941

> Handle gracefully an LZO-text table containing regular text files
> -
>
> Key: IMPALA-646
> URL: https://issues.apache.org/jira/browse/IMPALA-646
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 1.1.1
>Reporter: John Russell
>Priority: Minor
>  Labels: usability
>
> Somehow I managed to get Hive to create an LZO-text table yet not actually 
> compress the contents, so the data files did not have a .lzo suffix:
> hive> create table lzo_t (s string) STORED AS INPUTFORMAT 
> 'com.hadoop.mapred.DeprecatedLzoTextInputFormat' OUTPUTFORMAT 
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
> hive> describe formatted lzo_t;
> OK
> # col_name  data_type   comment
> s   string  None
> # Detailed Table Information
> Database:   default
> Owner:  cloudera
> CreateTime: Mon Oct 28 15:51:42 EDT 2013
> LastAccessTime: UNKNOWN
> Protect Mode:   None
> Retention:  0
> Location:   hdfs://127.0.0.1:8020/user/hive/warehouse/lzo_t
> Table Type: MANAGED_TABLE
> Table Parameters:
> numFiles2
> numPartitions   0
> numRows 0
> rawDataSize 0
> totalSize   139667
> transient_lastDdlTime   1382990204
> # Storage Information
> SerDe Library:  org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> InputFormat:com.hadoop.mapred.DeprecatedLzoTextInputFormat
> OutputFormat:   
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets:-1
> Bucket Columns: []
> Sort Columns:   []
> Storage Desc Params:
> serialization.format1
> Time taken: 4.479 seconds
> hive> insert into table lzo_t select s_manager from store;
> hive> insert into table lzo_t select s from parquet.billion_text limit 1;
> $ hdfs dfs -ls /user/hive/warehouse/lzo_t
> Found 2 items
> -rw-r--r--   1 cloudera supergroup156 2013-10-28 15:52 
> /user/hive/warehouse/lzo_t/00_0
> -rw-r--r--   1 cloudera supergroup 139511 2013-10-28 15:56 
> /user/hive/warehouse/lzo_t/00_0_copy_1
> $ hdfs dfs -cat /user/hive/warehouse/lzo_t/00_0
> William Ward
> Scott Smith
> Scott Smith
> Edwin Adams
> Edwin Adams
> Edwin Adams
> David Thomas
> Brett Yates
> Brett Yates
> Raymond Jacobs
> The data should be compressed, but it's not.
> Impala fails like so trying to access the table:
> [localhost:21000] > describe formatted lzo_t;
> Query: describe formatted lzo_t
> ERROR: AnalysisException: Failed to load metadata for table: default.lzo_t
> CAUSED BY: TableLoadingException: TableLoadingException: Failed to load 
> metadata for table: lzo_t
> CAUSED BY: RuntimeException: Expected file with .lzo suffix: 
> hdfs://127.0.0.1:8020/user/hive/warehouse/lzo_t/00_0
> com.cloudera.impala.catalog.TableLoadingException: Failed to load metadata 
> for table: lzo_t
> at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:776)
> at com.cloudera.impala.catalog.Table.load(Table.java:174)
> at com.cloudera.impala.catalog.Db.loadTable(Db.java:96)
> at com.cloudera.impala.catalog.Db.access$000(Db.java:53)
> at com.cloudera.impala.catalog.Db$1.load(Db.java:70)
> at com.cloudera.impala.catalog.Db$1.load(Db.java:66)
> at 
> com.cloudera.impala.catalog.CatalogObjectCache$1.call(CatalogObjectCache.java:229)
> at 
> com.cloudera.impala.catalog.CatalogObjectCache$1.call(CatalogObjectCache.java:226)
> at 
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
> at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at 
> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at 
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at 
> com.cloudera.impala.catalog.CatalogObjectCache.get(CatalogObjectCache.java:226)
> at com.cloudera.impala.catalog.Db.getTable(Db.java:190)
> at

[jira] [Resolved] (IMPALA-779) Incompatible type error when querying file created from AvroParquetWriter.

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-779.
--
Resolution: Duplicate

> Incompatible type error when querying file created from AvroParquetWriter.
> --
>
> Key: IMPALA-779
> URL: https://issues.apache.org/jira/browse/IMPALA-779
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 1.2.1
> Environment: CDH4.3
> Impala 1.2.1
>Reporter: Doug Meil
>Priority: Minor
>  Labels: usability
>
> Scenario:  
> 1) Created Parquet file with AvroParquetWriter in code with 100 or so columns.
> 2) Created external table with Parquet against this file defined with only 
> the first 4 columns and queried them all successfully.
> 3) Created second external table against this same file that was defined with 
> the last 4 columns and the query blows up - complaining about the first 
> column, and that wasn't even in the table definition.
> {code}
> [rd-namenode.explorys:21000] > select * from mytable2 limit 4;
> Query: select * from mytable2 limit 4
> ERROR: File 
> hdfs://namenode:8021/user/doug.meil/parquet/mytable/regid=2/myfile.prq has an 
> incompatible type with the table schema for column long1.  Expected type: 
> BYTE_ARRAY.  Actual type: INT64
> ERROR: Invalid query handle
> {code}
> The original Avro schema defined 'long1' like this...
> {code}
> {"name": "long1", "type": "long"},
> {code}
> The fact that the "Actual type" is INT64 seems correct - because I meant to 
> put a long in there.  Why does Impala think the expected type is a BYTE_ARRAY?
> Note:  summary queries (e.g., select count(*) from mytable2) actually WORK.  
> Go figure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-781) Impala cannot read data written by using the LazyBinaryColumnarSerDe (Hive's new default SerDe for RC file in Hive v.12)

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-781.
--
Resolution: Won't Do

I don't think we're likely to invest effort at this point into improving RCFile 
support, given that Parquet is now the standard.

> Impala cannot read data written by using the LazyBinaryColumnarSerDe (Hive's 
> new default SerDe for RC file in Hive v.12)
> 
>
> Key: IMPALA-781
> URL: https://issues.apache.org/jira/browse/IMPALA-781
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 1.2.3
>Reporter: Lenni Kuff
>Priority: Minor
>
> Impala cannot read data written by using the LazyBinaryColumnarSerDe (Hive's 
> new default SerDe for RC file in Hive v.12). For the short term we need to 
> add an analysis check to say this is not supported, but we should also update 
> our RCFile scanner to support the new format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-1051) Impala should maintain the user identifier rather than lower casing everything

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1051.
---
Resolution: Won't Fix

I don't think we're likely to change this behaviour.

> Impala should maintain the user identifier rather than lower casing everything
> --
>
> Key: IMPALA-1051
> URL: https://issues.apache.org/jira/browse/IMPALA-1051
> Project: IMPALA
>  Issue Type: Bug
>  Components: Catalog
>Affects Versions: Impala 1.3.1
>Reporter: Nong Li
>Priority: Minor
>  Labels: catalog-server, usability
>
> We convert things to lower case pretty early and while that simplifies some 
> of our internal logic is not as good for the users. 
> One example is column names. converting it to lower case hurts readability 
> and interoperability with tools that read the files generated by impala (i.e. 
> for parquet).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-1081) String expression discards data when single quote inside double-quoted string

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1081.
---
Resolution: Cannot Reproduce

I can't reproduce this on a recent build. I think IMPALA-3942 probably fixed it.

> String expression discards data when single quote inside double-quoted string
> -
>
> Key: IMPALA-1081
> URL: https://issues.apache.org/jira/browse/IMPALA-1081
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 1.4
>Reporter: John Russell
>Assignee: Alexander Behm
>Priority: Major
>
> If I use INSERT...VALUES with a double-quoted string, and the string contains 
> any single quotes, some of the string data is lost.
> [localhost:21000] > create table single_quote (s string);
> [localhost:21000] > insert into single_quote values ("I'm just tryin' to 
> insert a value where I'm using single quotes.");
> [localhost:21000] > select * from single_quote;
> +-+
> | s   |
> +-+
> | I to insert a value where I |
> +-+
> What I have observed is that everything after the first single quote is 
> discarded. (I.e. if there's only one single quote, everything after that is 
> truncated.) If there's another single quote later, the text following that is 
> preserved, at least until the next single quote when it's discarded again.
> The quotes and the discarded text are actually missing from the table 
> contents when I look at the text data file.
> Things I didn't check:
> - Whether this affects INSERT...SELECT or only INSERT...VALUES.
> - If it happens with inserts via JDBC.
> It looks like it's something directly in expression evaluation for strings, 
> because the data gets thrown away from any string expression:
> [localhost:21000] > select "I'm only a squirrel trying to get a nut";
> +---+
> | 'i'm only a squirrel trying to get a nut' |
> +---+
> | I |
> +---+
> [localhost:21000] > select "I'm hungry" = "I'm thirsty";
> +--+
> | 'i'm hungry' = 'i'm thirsty' |
> +--+
> | true |
> +--+
> Assigning to Abdullah because he and I and Ishaan talked about it maybe being 
> an impala-shell bug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7146) log session ID in Impala daemon log

2018-06-07 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7146.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> log session ID in Impala daemon log
> ---
>
> Key: IMPALA-7146
> URL: https://issues.apache.org/jira/browse/IMPALA-7146
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Distributed Exec
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>  Labels: supportability
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> We should log the session_id to query_id mapping to enable debugging
> This would be very useful for investigating issues with idle_session_timeout, 
> etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-1926) Cancelling query in impala-shell suppresses further informational messages

2018-06-07 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1926.
---
Resolution: Cannot Reproduce

I don't know if we've seen this for a while. The code has changed a lot, e.g. 
with IMPALA-1144

> Cancelling query in impala-shell suppresses further informational messages
> --
>
> Key: IMPALA-1926
> URL: https://issues.apache.org/jira/browse/IMPALA-1926
> Project: IMPALA
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: Impala 2.2
> Environment: 2.2 pre-release
>Reporter: John Russell
>Assignee: Ishaan Joshi
>Priority: Minor
>  Labels: shell
>
> I cancelled an INSERT statement when I realized it was going to take a long 
> time and not match anything.
> {code}
> [localhost:21000] > insert into partitioned_text (x, y, s) partition 
> (year=2014,month=1,day=1) select id, val, name from oreilly.sample_data
>   > where zerofill like '20140101%' limit 1e6;
> ^C Cancelling Query
> Application Exception : Default TException.
> {code}
> Subsequent INSERT statements did not show any 'Inserted NNN rows' messages, 
> so I assumed they were unsuccessful.
> {code}
> [localhost:21000] > insert into partitioned_text (x, y, s) partition 
> (year=2014,month=1,day=1) select id, val, name from oreilly.sample_data
> where id = 20140101 limit 1e6;
> [localhost:21000] > insert into partitioned_text (x, y, s) partition 
> (year=2014,month=1,day=1) select id, val, name from oreilly.normalized_parquet
> where id between 1 and 100; 
> {code}
> However, they apparently did succeed and inserted the expected number of rows.
> {code}
> [localhost:21000] > select count(*) from partitioned_text;
> +--+
> | count(*) |
> +--+
> | 101  |
> +--+
> {code}
> My hypothesis is that the error handler in impala-shell turned off some 
> messaging flag when I cancelled the query via Ctrl-C, and this flag remained 
> off afterwards.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-7062) Fix unsafe RuntimeProfile::SortChildren() API

2018-06-07 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-7062.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

> Fix unsafe RuntimeProfile::SortChildren() API
> -
>
> Key: IMPALA-7062
> URL: https://issues.apache.org/jira/browse/IMPALA-7062
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> As a follow-up to IMPALA-6484, we should fix the broken SortChildren() API 
> that is not thread-safe and dangerous. We should replace it with something 
> that sorts the children by total time using stable values that can't be 
> modified concurrently by other threads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-1904) Issue with Impala JDBC using Spring JDBC Template with pooled datasource

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1904.
---
Resolution: Invalid

We don't track bugs for this client here, it's not part of Apache IMpala.

> Issue with Impala JDBC using Spring JDBC Template with pooled datasource
> 
>
> Key: IMPALA-1904
> URL: https://issues.apache.org/jira/browse/IMPALA-1904
> Project: IMPALA
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: Impala 2.1.1
> Environment: Wiindows 7 64bit, java 6 & 7
>Reporter: Sandeep
>Assignee: Syed A. Hashmi
>Priority: Minor
>  Labels: impala
> Attachments: TestWeb1.zip
>
>
> Hi,
> I am facing a strange issue while using imapala jdbc drivers to connect with 
> hadoop.
> The scenario is when UI is hitting two of my REST services which internally 
> trying to fetch data from 3 tables, one of the API throws exception saying 
> SQL DataIntegrityViolation. This is a random behaviour from both the API's. 
> Both the API works fine if we call them one by one. This is like UI is making 
> async calls at the same time.
> I am using Spring framework and using JDBCTemplate to run sql statements. We 
> have a Single instance of DataSource throughout the application.  No matter 
> whether I use Imapala's DataSource which supports pooling or Spring 
> DriverManagerDataSource result is same. On the other side it works fine if i 
> use Core JDBC syntax like closing statement and connection object after each 
> call. In Spring closing of statement and connection object is being handled 
> by Spring JDBCTemplate itself.
> Please let me know if its a known issue and we have an alternate for that.
> Thanks
> Sandeep Verma



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2156) Specify and test full lifecycle for expressions

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2156.
---
Resolution: Fixed

I think IMPALA-4192 addressed this.

> Specify and test full lifecycle for expressions
> ---
>
> Key: IMPALA-2156
> URL: https://issues.apache.org/jira/browse/IMPALA-2156
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.2
>Reporter: Henry Robinson
>Priority: Minor
>
> {{expr-context.h}} does not fully specify what methods on an {{ExprContext}} 
> are legal to call, when. For example, is it legal to {{Close()}} an 
> {{ExprContext}} that has not yet been {{Prepare()}}'d? Can we call {{Open()}} 
> twice? What state is copied when {{Clone()}} is called?
> Without this, it's hard to write robust error handling in objects that use 
> {{ExprContext}} instances. It is also hard to write expressions that are 
> correct in all uses. 
> We should update the comments in {{expr-context.h}} to define the invariants 
> required of and guarantees made to {{ExprContext}} clients. We can then add 
> tests that exercise the legal lifecycles of all expressions to root out any 
> bugs, and add invariant checks in the code that will catch bad clients.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2176) Inconsistency in the handling of SET commands by the Impala shell

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2176.
---
Resolution: Cannot Reproduce

> Inconsistency in the handling of SET commands by the Impala shell
> -
>
> Key: IMPALA-2176
> URL: https://issues.apache.org/jira/browse/IMPALA-2176
> Project: IMPALA
>  Issue Type: Bug
>  Components: Clients
>Affects Versions: Impala 2.3.0
>Reporter: Andre Araujo
>Priority: Minor
>  Labels: shell
>
> The Impala shell handle SET commands in two different ways:
> * Locally, if the line starts with the keyword SET (e.g. "SET 
> EXPLAIN_LEVEL=3")
> * As a query, if the lines starts with a comment (e.g. "-- dummy comment\nSET 
> EXPLAIN_LEVEL=3")
> In the first case above the shell keeps the state of the set options locally 
> and passes that to the Beeswax service as a configuration object. In the 
> second case, the whole statement (comments included) are sent to the Impala 
> frontend as a query.
> The inconsistency in handling SET statements, causes the Impala shell local 
> configuration to go out-of-sync with the Impala frontend's, as shown below:
> {code}
> [poti:21000] > set explain_level=3;
> EXPLAIN_LEVEL set to 3
> [poti:21000] > -- this one executes as a query
>  > set explain_level=2;
> Query: --
> set explain_level=2
> Fetched 0 row(s) in 0.27s
> [poti:21000] > set;
> Query options (defaults shown in []):
> ...
>   EXPLAIN_LEVEL: 3
> ...
> [poti:21000] > -- again, this executes as a query
>  > set;
> Query: --
> set
> +-+---+
> | option  | value |
> +-+---+
> ...
> | EXPLAIN_LEVEL   | 2 |
> ...
> +-+---+
> Fetched 31 row(s) in 0.01s
> [poti:21000] > 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-882) Accept "hbase.table.default.storage.type" in both TBLPROPERTIES and SERDEPROPERTIES

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-882.
--
Resolution: Won't Fix

Closing due to insufficient interest.

> Accept "hbase.table.default.storage.type" in both TBLPROPERTIES and 
> SERDEPROPERTIES
> ---
>
> Key: IMPALA-882
> URL: https://issues.apache.org/jira/browse/IMPALA-882
> Project: IMPALA
>  Issue Type: Bug
>  Components: Catalog
>Affects Versions: Impala 1.0.1
>Reporter: Alan Choi
>Priority: Minor
>  Labels: catalog-server, usability
>
> Need to figure out if we should support this property in both places. It's 
> clearly a table property. However, Hive accepted it in both.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-1373) Snappy writer includes extra bytes at end

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1373.
---
Resolution: Cannot Reproduce

> Snappy writer includes extra bytes at end
> -
>
> Key: IMPALA-1373
> URL: https://issues.apache.org/jira/browse/IMPALA-1373
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.0
>Reporter: John Russell
>Priority: Minor
>
> Marking as minor because file writer functionality not part of the 2.0 
> release.
> Discussed with Ippo, he has a fix ready to go.
> If Impala writes a .snappy file, some extra bytes might get included by 
> mistake:
> {code}
> [localhost:21000] > set compression_codec=snappy;
> [localhost:21000] > set allow_unsupported_formats=1;
> [localhost:21000] > insert into csv_compressed values
> ('one - snappy','two - snappy','three - snappy');
> [localhost:21000] > select * from csv_compressed;
> +++--+
> | a  | b  | c|
> +++--+
> | one - snappy   | two - snappy   | three - snappy   |
> | s0��| NULL   |  |
> | one - uncompressed | two - uncompressed | three - uncompressed |
> | abc - uncompressed | xyz - uncompressed | 123 - uncompressed   |
> | one - bz2  | two - bz2  | three - bz2  |
> | abc - bz2  | xyz - bz2  | 123 - bz2|
> | one - gzip | two - gzip | three - gzip |
> | abc - gzip | xyz - gzip | 123 - gzip   |
> +++--+
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-1428) Subquery: "...WHERE (SELECT 1) > 1": InternalException: Required field 'input_row_tuples' was not present

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-1428.
---
Resolution: Cannot Reproduce

> Subquery: "...WHERE (SELECT 1) > 1": InternalException: Required field 
> 'input_row_tuples' was not present
> -
>
> Key: IMPALA-1428
> URL: https://issues.apache.org/jira/browse/IMPALA-1428
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 2.1, Impala 2.2
>Reporter: casey
>Priority: Minor
>  Labels: planner, ramp-up
>
> As far as I can tell this only happens when the subquery has no FROM clause. 
> I think someone should at least investigate to see if there is a more common 
> use case repro.
> Top query shows the error, bottom query works fine.
> {noformat}
> Query: select 1 FROM alltypestiny WHERE (SELECT 1) > 1
> ERROR: InternalException: Required field 'input_row_tuples' was not present! 
> Struct: TExchangeNode(input_row_tuples:null)
> [casey-desktop:21000] > SELECT 1 FROM alltypestiny WHERE (SELECT COUNT(*) 
> FROM alltypestiny) > 1;
> Query: select 1 FROM alltypestiny WHERE (SELECT COUNT(*) FROM alltypestiny) > 
> 1
> +---+
> | 1 |
> +---+
> | 1 |
> | 1 |
> | 1 |
> | 1 |
> | 1 |
> | 1 |
> | 1 |
> | 1 |
> +---+
> Fetched 8 row(s) in 0.87s
> {noformat}
> stack trace 
> {noformat}
> I1027 11:42:41.703457 25243 jni-util.cc:177] 
> com.cloudera.impala.common.InternalException: Required field 
> 'input_row_tuples' was not present! Struct: 
> TExchangeNode(input_row_tuples:null)
>   at 
> com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:147)
>   
> I1027 11:42:41.734680 25243 status.cc:44] InternalException: Required field 
> 'input_row_tuples' was not present! Struct: 
> TExchangeNode(input_row_tuples:null)
> @   0xe87cf6  impala::Status::Status()
> 
> @  0x10eb397  impala::JniUtil::GetJniExceptionMsg()   
> 
> @   0xf6ea4c  impala::JniUtil::CallJniMethod<>()  
> 
> @   0xf6b695  impala::Frontend::GetExecRequest()  
> 
> @   0xf8bd02  impala::ImpalaServer::ExecuteInternal() 
> 
> @   0xf8b887  impala::ImpalaServer::Execute() 
> 
> @  0x102794a  impala::ImpalaServer::query()   
> 
> @  0x127487b  beeswax::BeeswaxServiceProcessor::process_query()   
> 
> @  0x127458c  beeswax::BeeswaxServiceProcessor::dispatchCall()
> 
> @  0x12510cf  impala::ImpalaServiceProcessor::dispatchCall()
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2346) Create unit test that exposes (now fixed) disk IO mgr race

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2346.
---
   Resolution: Fixed
Fix Version/s: Impala 3.1.0
   Impala 2.13.0

I added a test like this for IMPALA-4835

> Create unit test that exposes (now fixed) disk IO mgr race
> --
>
> Key: IMPALA-2346
> URL: https://issues.apache.org/jira/browse/IMPALA-2346
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.2.4
>Reporter: Skye Wanderman-Milne
>Priority: Minor
>  Labels: test
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> The bug fixed by 
> https://github.com/cloudera/Impala/commit/798007bef8a56e36f2d8e0d058ca820e15bb6ade
>  should have an io mgr backend test that exposes it. I don't think it's even 
> possible to expose it without 0-length scan ranges 
> (http://gerrit.cloudera.org:8080/#/c/711/), but even the test added in that 
> patch doesn't expose this bug. I think we may need to mix empty and non-empty 
> scan ranges in order to expose it in a test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2600) TestScratchDir.test_multiple_dirs should contain enough information to debug failures when impalad.INFO is unavailable

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2600.
---
Resolution: Fixed

> TestScratchDir.test_multiple_dirs should contain enough information to debug 
> failures when impalad.INFO is unavailable
> --
>
> Key: IMPALA-2600
> URL: https://issues.apache.org/jira/browse/IMPALA-2600
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.5.0
>Reporter: Jim Apple
>Priority: Critical
>  Labels: test, test-infra
>
> Impala.tests.custom_cluster.test_scratch_disk.TestScratchDir.test_multiple_dirs
>  failed in http://sandbox.jenkins.cloudera.com/job/impala-s3/279/ at the line 
> where it expects a certain string to be in impalad.INFO.  It is difficult to 
> debug such a test if it does not include enough information to debug what was 
> in impalad.INFO at the time of the failure, especially if the log either 
> disappears or, after the test fails, includes the string in question.
> See also https://issues.cloudera.org/browse/IMPALA-2574.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2561) Functional test coverage for parquet files with invalid data

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2561.
---
Resolution: Fixed

We have a bunch of bad files checked in now.

> Functional test coverage for parquet files with invalid data
> 
>
> Key: IMPALA-2561
> URL: https://issues.apache.org/jira/browse/IMPALA-2561
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.3.0
>Reporter: Tim Armstrong
>Priority: Minor
>  Labels: test
>
> IMPALA-2558 ran into a DCHECK while reading a parquet file with a bad block. 
> We need tests to cover the cases when we're reading a parquet file with some 
> invalid data not in the header or footer, e.g. the def/rep levels.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2827) Frontend does not compile with Thrift 0.9.3

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2827.
---
Resolution: Duplicate

> Frontend does not compile with Thrift 0.9.3
> ---
>
> Key: IMPALA-2827
> URL: https://issues.apache.org/jira/browse/IMPALA-2827
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.5.0
>Reporter: Martin Grund
>Priority: Minor
>  Labels: osx
>
> Impala on Mac cannot use Thrift 0.9.0 and we have to move to a higher 
> version. The latest release is 0.9.3 and for unkown reasons this breaks 
> compilation of our Java frontend.
> The compile error can be very easily fixed by removing some of the generated 
> code that tries to Implement an asynchronous interface, however the root 
> cause seems to be something else. It probably is related to the interface 
> that is provided by Hive for the HS2 interface.
> {code}
> [INFO] -
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] 
> /Users/mgrund/Impala/fe/generated-sources/gen-java/com/cloudera/impala/thrift/ImpalaHiveServer2Service.java:[264,114]
>  cannot find symbol
>   symbol:   class AsyncProcessor
>   location: class org.apache.hive.service.cli.thrift.TCLIService
> [INFO] 1 error
> {code}
> Digging deeper the reason is pretty simple, that the Hive code that we use to 
> Import the {{TCLIService}} does simply not contain the async part since it 
> was not generated with Thrift 0.9.3.
> Here is the necessary code in the Impala build:
> {code}
> IF (FIL STREQUAL "TCLIService.thrift" OR FIL STREQUAL "parquet.thrift" OR
> FIL STREQUAL "cli_service.thrift")
>   # HiveServer2 and Parquet thrift generation
>   # Do not generate Java source because we should use the jar from Hive 
> or Parquet
>   add_custom_command(
> OUTPUT ${OUTPUT_BE_FILE}
> COMMAND ${THRIFT_COMPILER} ${CPP_ARGS} ${FIL}
> COMMAND ${THRIFT_COMPILER} ${PYTHON_ARGS} ${FIL}
> DEPENDS ${ABS_FIL}
> COMMENT "Running thrift compiler on ${FIL}"
> VERBATIM
>   )
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2714) metadata.test_ddl.TestDdlStatements.test_functions_ddl fails on repeated runs

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2714.
---
Resolution: Fixed

commit ab9e54bc4298549048b1935837e8e818d5b980ce
Author: Alex Behm 
Date:   Tue May 10 01:01:32 2016 -0700

IMPALA-3491: Use unique database fixture in test_ddl.py.

Adds new parametrization to the unique database fixture:
- num_dbs: allows creating multiple unique databases at once;
  the 2nd, 3rd, etc. datbase name is generated by appending
  "2", "3", etc., to the first database name
- sync_ddl: allows creating the dabatases(s) with sync_ddl
  which is needed by most tests in test_ddl.py

Testing: I ran debug/core and debug/exhaustive on HDFS and
core/debug on S3. Also ran the test locally in a loop on
exhaustive.

Change-Id: Idf667dd5e960768879c019e2037cf48ad4e4241b
Reviewed-on: http://gerrit.cloudera.org:8080/4155
Reviewed-by: Alex Behm 
Tested-by: Internal Jenkins



> metadata.test_ddl.TestDdlStatements.test_functions_ddl fails on repeated runs
> -
>
> Key: IMPALA-2714
> URL: https://issues.apache.org/jira/browse/IMPALA-2714
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.3.0
>Reporter: Harrison Sheinblatt
>Priority: Minor
>  Labels: test_issue
>
> The test fails when run a second time without restarting impalad because it 
> uses the default database and doesn't clean up udfs.
> {noformat}
> metadata.test_ddl.TestDdlStatements.test_functions_ddl[exec_option: 
> {'batch_size': 0, 'num_nodes': 0, 'sync_ddl': 0, 'disable_codegen': False, 
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: 
> text/none] (from pytest)
> {noformat}
> Test log:
> {noformat}
> Error Message
> assert Comparing QueryTestResults (expected vs actual):   'INT','fn()' == 
> 'INT','fn()'   None != 'STRING','udf_update_test_create1()'   None != 
> 'STRING','udf_update_test_create2()'   None != 
> 'STRING','udf_update_test_drop()'   Number of rows returned (expected vs 
> actual): 1 != 4
> Stacktrace
> metadata/test_ddl.py:347: in test_functions_ddl
> multiple_impalad=self._use_multiple_impalad(vector))
> common/impala_test_suite.py:235: in run_test_case
> pytest.config.option.update_results)
> common/test_result_verifier.py:345: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:197: in verify_query_result_is_equal
> assert expected_results == actual_results
> E   assert Comparing QueryTestResults (expected vs actual):
> E 'INT','fn()' == 'INT','fn()'
> E None != 'STRING','udf_update_test_create1()'
> E None != 'STRING','udf_update_test_create2()'
> E None != 'STRING','udf_update_test_drop()'
> E Number of rows returned (expected vs actual): 1 != 4
> Standard Error
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `ddl_test_db` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `ddl_purge_db` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `alter_table_test_db` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `alter_table_test_db2` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `function_ddl_test` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `udf_test` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `data_src_test` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `truncate_table_test_db` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `test_db` cascade;
> -- executing against localhost:21000
> use default;
> SET sync_ddl=1;
> -- executing against localhost:21000
> drop database if exists `alter_purge_db` cascade;
> MainThread: Starting new HTTP connection (1): 0.0.0.0
> MainThread: Starting new HTTP connection (1): 0.0.0.0
> MainThread: Starting new HTTP connection (1): 0.0.0.0
> -- connecting to: localhost:21000
> SET sync_ddl=1;
> -- executing against

[jira] [Resolved] (IMPALA-3623) Triage backend code coverage issues

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-3623.
---
Resolution: Later

> Triage backend code coverage issues
> ---
>
> Key: IMPALA-3623
> URL: https://issues.apache.org/jira/browse/IMPALA-3623
> Project: IMPALA
>  Issue Type: Task
>  Components: Backend
>Affects Versions: Impala 2.6.0
>Reporter: Tim Armstrong
>Priority: Major
>
> Go through backend code coverage reports and identify any missing test 
> coverage or dead code that should be fixed. Ideally also figure out a way to 
> exclude code that we don't expect to be executed.
> I took an initial pass and there are a few categories of missing coverage 
> that is expected to be missing:
> * benchmark binaries that aren't run
> * deprecated features
> * DCHECKs that don't fire (only one arm of the branch is taken according to 
> branch coverage metrics)
> * debugging code like DebugString() methods
> * Code from the UDF development kit
> * Old agg and join code



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-2879) Jenkins job compute-stats-10node-cdh5 needs to be more robust.

2018-06-11 Thread Tim Armstrong (JIRA)



 [ 
https://issues.apache.org/jira/browse/IMPALA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-2879.
---
Resolution: Invalid

Not relevant to Apache Impala project

> Jenkins job compute-stats-10node-cdh5 needs to be more robust.
> --
>
> Key: IMPALA-2879
> URL: https://issues.apache.org/jira/browse/IMPALA-2879
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.3.0
>Reporter: Alexander Behm
>Assignee: Mostafa Mokhtar
>Priority: Critical
>
> There are several problems with the following job (and the secure one):
> http://sandbox.jenkins.cloudera.com/view/Impala/view/Cluster%20Runs/view/10-node-cdh5/job/compute-stats-10node-cdh5/
> The job relies on checking out and building the Impala repo. Not only is this 
> unnecessary, it also makes the job slow (only 40% of the time spent actually 
> computing stats) ad brittle. For example, I just had to debug a problem 
> caused by an update to the Impala toolchain. The change in the toolchain 
> seems completely unrelated to the goal of this job. Our perf runs blindly run 
> the compute stats job before any queries, so breaking this job means our perf 
> runs fail.
> We should clean this up to not rely on the Impala repo and make this job more 
> reliable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 7585 matches

Mail list logo