[jira] [Resolved] (IMPALA-6485) BE compilation failure: error: ‘EVP_CTRL_GCM_SET_IVLEN’ was not declared in this scope

2018-02-06 Thread Alexander Behm (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Behm resolved IMPALA-6485.

   Resolution: Fixed
Fix Version/s: Impala 2.12.0

commit 4ca7f261e437c2eb49d8d114e9af3696c61428f4
Author: Alex Behm 
Date:   Tue Feb 6 11:50:51 2018 -0800

Revert "IMPALA-6219: Use AES-GCM for spill-to-disk encryption"

This reverts commit 9b68645f9eb9e08899fda860e0946cc05f205479.

Change-Id: Ia06f061a4ecedd1df0d359fe06fe84618b5e07fb


> BE compilation failure: error: ‘EVP_CTRL_GCM_SET_IVLEN’ was not declared in 
> this scope
> --
>
> Key: IMPALA-6485
> URL: https://issues.apache.org/jira/browse/IMPALA-6485
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.12.0
>Reporter: Alexander Behm
>Assignee: Tim Armstrong
>Priority: Blocker
>  Labels: broken-build
> Fix For: Impala 2.12.0
>
>
> Failure:
> {code}
> 20:39:32 
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:
>  In member function ‘impala::Status 
> impala::EncryptionKey::EncryptInternal(bool, const uint8_t*, int64_t, 
> uint8_t*)’:
> 20:39:32 
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:134:31:
>  error: ‘EVP_CTRL_GCM_SET_IVLEN’ was not declared in this scope
> 20:39:32  EVP_CIPHER_CTX_ctrl(, EVP_CTRL_GCM_SET_IVLEN, 
> AES_BLOCK_SIZE, NULL);
> 20:39:32^
> 20:39:32 
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:169:31:
>  error: ‘EVP_CTRL_GCM_SET_TAG’ was not declared in this scope
> 20:39:32  EVP_CIPHER_CTX_ctrl(, EVP_CTRL_GCM_SET_TAG, AES_BLOCK_SIZE, 
> gcm_tag_);
> 20:39:32^
> 20:39:32 
> /data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:181:31:
>  error: ‘EVP_CTRL_GCM_GET_TAG’ was not declared in this scope
> 20:39:32  EVP_CIPHER_CTX_ctrl(, EVP_CTRL_GCM_GET_TAG, AES_BLOCK_SIZE, 
> gcm_tag_);
> 20:39:32^
> 20:39:32 make[2]: *** [be/src/util/CMakeFiles/Util.dir/openssl-util.cc.o] 
> Error 1
> 20:39:32 make[2]: *** Waiting for unfinished jobs
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-5037) Change default Parquet array resolution according to Parquet standard.

2018-02-06 Thread Alexander Behm (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Behm resolved IMPALA-5037.

   Resolution: Fixed
Fix Version/s: Impala 3.0

commit dedff5e25c19da724ae032a52354cd4fd61ec40a
Author: Alex Behm 
Date:   Thu Feb 1 16:57:53 2018 -0800

IMPALA-5037: Default PARQUET_ARRAY_RESOLUTION=THREE_LEVEL

Changes the default value for the PARQUET_ARRAY_RESOLUTION
query option to conform to the Parquet standard.
Before: TWO_LEVEL_THEN_THREE_LEVEL
After:  THREE_LEVEL

For more information see:
* IMPALA-4725
* https://github.com/apache/parquet-format/blob/master/LogicalTypes.md

Testing:
- expands and cleans up the existing tests for more coverage
  over the different resolution policies
- private core/hdfs run passed

Cherry-picks: not for 2.x.

Change-Id: Ib8f7e9010c4354d667305d9df7b78862efb23fe1
Reviewed-on: http://gerrit.cloudera.org:8080/9210
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins


> Change default Parquet array resolution according to Parquet standard.
> --
>
> Key: IMPALA-5037
> URL: https://issues.apache.org/jira/browse/IMPALA-5037
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Affects Versions: Impala 2.8.0, Impala 2.9.0
>Reporter: Alexander Behm
>Assignee: Alexander Behm
>Priority: Major
>  Labels: include-in-v3, incompatibility
> Fix For: Impala 3.0
>
>
> With IMPALA-4725 we've introduced query options to control the field 
> resolution behavior when scanning Parquet files with nested arrays. The 
> current default strategy currently tries to auto-detect the array encoding 
> within Parquet files, but this strategy can sometimes subtly go wrong and 
> return incorrect results due to the inherent ambiguity of the 2/3-level 
> encoding schemes in Parquet.
> We should switch the default resolution strategy according to the Parquet 
> standard 3-level encoding, instead of the current auto-detect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (IMPALA-6473) Error in analytic sort with same expr in 'partition by' and 'order by'

2018-02-06 Thread Thomas Tauber-Marshall (JIRA) 

 [ 
https://issues.apache.org/jira/browse/IMPALA-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Tauber-Marshall resolved IMPALA-6473.

   Resolution: Fixed
Fix Version/s: Impala 3.0

commit 32bf54dfd03c1f880864ff6295df50a921dd5922 (HEAD -> analytic-sort, 
asf-gerrit/master)
Author: Thomas Tauber-Marshall 
Date:   Mon Feb 5 14:24:23 2018 -0800

IMPALA-6473: Fix analytic fn that partitions and orders on same expr

Previously, an analytic function that used the same expr in both the
'partition by' and 'order by' clauses, and where the expr meets the
criteria for being materialized before the sort, would hit an
IllegalStateException due to the expr being inserted into the same
ExprSubstitutionMap twice.

If the values have already been partitioned on the expr, then all of
the values for it in each partition will be the same and also ordering
on the expr doesn't change the results. So, the fix is to simply
exclude the duplicate expr from the 'order by' while still
partitioning on it.

Testing:
- Added a regression test to PlannerTest.

Change-Id: Id5f1d5fbc6f69df5850f96afed345ce27668c30b
Reviewed-on: http://gerrit.cloudera.org:8080/9218
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins

> Error in analytic sort with same expr in 'partition by' and 'order by'
> --
>
> Key: IMPALA-6473
> URL: https://issues.apache.org/jira/browse/IMPALA-6473
> Project: IMPALA
>  Issue Type: Bug
>  Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Fix For: Impala 3.0
>
>
> Repro:
> {noformat}
> drop table if exists default.tt;
> CREATE TABLE default.tt (instruction_date timestamp);
> SELECT sum(1) OVER( partition by to_date(instruction_date) ORDER BY 
> to_date(instruction_date) rows UNBOUNDED PRECEDING) FROM default.tt;
> {noformat}
> Stack:
> {noformat}
> I0205 14:16:03.763309  4889 jni-util.cc:230] java.lang.IllegalStateException 
> at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:129) 
> at 
> org.apache.impala.analysis.ExprSubstitutionMap.verify(ExprSubstitutionMap.java:162)
>  
> at 
> org.apache.impala.analysis.ExprSubstitutionMap.combine(ExprSubstitutionMap.java:128)
>  
> at 
> org.apache.impala.planner.AnalyticPlanner.createSortInfo(AnalyticPlanner.java:304)
>  
> at 
> org.apache.impala.planner.AnalyticPlanner.createSortGroupPlan(AnalyticPlanner.java:348)
>  
> at 
> org.apache.impala.planner.AnalyticPlanner.createSingleNodePlan(AnalyticPlanner.java:120)
>  
> at 
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:269)
>  
> at 
> org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:150)
>  
> at org.apache.impala.planner.Planner.createPlan(Planner.java:98) 
> at 
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1030) 
> at 
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1127) 
> at 
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156) 
> {noformat}
> The problem is caused by using the same expr in both the 'partition by' and 
> 'order by' clauses of the analytic function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6485) BE compilation failure: error: ‘EVP_CTRL_GCM_SET_IVLEN’ was not declared in this scope

2018-02-06 Thread Alexander Behm (JIRA) 
Alexander Behm created IMPALA-6485:
--

 Summary: BE compilation failure: error: ‘EVP_CTRL_GCM_SET_IVLEN’ 
was not declared in this scope
 Key: IMPALA-6485
 URL: https://issues.apache.org/jira/browse/IMPALA-6485
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Alexander Behm
Assignee: Tim Armstrong


Failure:
{code}
20:39:32 
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:
 In member function ‘impala::Status 
impala::EncryptionKey::EncryptInternal(bool, const uint8_t*, int64_t, 
uint8_t*)’:
20:39:32 
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:134:31:
 error: ‘EVP_CTRL_GCM_SET_IVLEN’ was not declared in this scope
20:39:32  EVP_CIPHER_CTX_ctrl(, EVP_CTRL_GCM_SET_IVLEN, AES_BLOCK_SIZE, 
NULL);
20:39:32^
20:39:32 
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:169:31:
 error: ‘EVP_CTRL_GCM_SET_TAG’ was not declared in this scope
20:39:32  EVP_CIPHER_CTX_ctrl(, EVP_CTRL_GCM_SET_TAG, AES_BLOCK_SIZE, 
gcm_tag_);
20:39:32^
20:39:32 
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/openssl-util.cc:181:31:
 error: ‘EVP_CTRL_GCM_GET_TAG’ was not declared in this scope
20:39:32  EVP_CIPHER_CTX_ctrl(, EVP_CTRL_GCM_GET_TAG, AES_BLOCK_SIZE, 
gcm_tag_);
20:39:32^
20:39:32 make[2]: *** [be/src/util/CMakeFiles/Util.dir/openssl-util.cc.o] Error 
1
20:39:32 make[2]: *** Waiting for unfinished jobs
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6484) Crash in impala::RuntimeProfile::SortChildren

2018-02-06 Thread Alexander Behm (JIRA) 
Alexander Behm created IMPALA-6484:
--

 Summary: Crash in impala::RuntimeProfile::SortChildren
 Key: IMPALA-6484
 URL: https://issues.apache.org/jira/browse/IMPALA-6484
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.12.0
Reporter: Alexander Behm
Assignee: Lars Volker


Lars, assigning to you since you touched the relevant code last

Tests running at the time (as far as I can tell):
{code}
query_test/test_spilling.py
query_test/test_insert_parquet.py
TestScannersFuzzing.test_fuzz_decimal_tbl
{code}

Backtrace:
{code}
#0  0x0030120328e5 in raise () from /lib64/libc.so.6
#1  0x0030120340c5 in abort () from /lib64/libc.so.6
#2  0x7f49030c81a5 in os::abort(bool) () from 
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#3  0x7f4903258843 in VMError::report_and_die() () from 
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#4  0x7f49030cd562 in JVM_handle_linux_signal () from 
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#5  0x7f49030c44f3 in signalHandler(int, siginfo*, void*) () from 
/opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
#6  
#7  0x0182355a in std::_Rb_tree, 
std::pair const, impala::RuntimeProfile::Counter*>, 
std::_Select1st const, impala::RuntimeProfile::Counter*> >, 
std::less >, std::allocator const, impala::RuntimeProfile::Counter*> > >::_M_begin 
(this=0x382e657461745393) at 
/data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_tree.h:518
#8  0x01821eb2 in std::_Rb_tree, 
std::pair const, impala::RuntimeProfile::Counter*>, 
std::_Select1st const, impala::RuntimeProfile::Counter*> >, 
std::less >, std::allocator const, impala::RuntimeProfile::Counter*> > 
>::lower_bound (this=0x382e657461745393, __k=...) at 
/data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_tree.h:927
#9  0x01820bc9 in std::map, 
impala::RuntimeProfile::Counter*, std::less >, 
std::allocator const, impala::RuntimeProfile::Counter*> > 
>::lower_bound (this=0x382e657461745393, __x=...) at 
/data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_map.h:902
#10 0x0181f8b6 in std::map, 
impala::RuntimeProfile::Counter*, std::less >, 
std::allocator const, impala::RuntimeProfile::Counter*> > >::operator[] 
(this=0x382e657461745393, __k=...) at 
/data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_map.h:496
#11 0x0184e526 in impala::RuntimeProfile::total_time_counter 
(this=0x382e657461745373) at 
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/util/runtime-profile.h:261
#12 0x02cb656b in operator() (this=0x7f483784a380, a=..., b=...) at 
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/be/src/runtime/coordinator-backend-state.cc:554
#13 0x02cbe794 in 
__gnu_cxx::__ops::_Val_comp_iter::operator(), __gnu_cxx::__normal_iterator*, 
std::vector > > > 
(this=0x7f483784a380, __val=..., __it=...) at 
/data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/predefined_ops.h:166
#14 0x02cbdcc0 in 
std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator*, std::vector > >, 
__gnu_cxx::__ops::_Val_comp_iter > (__last=..., __comp=...) 
at 
/data/jenkins/workspace/impala-asf-master-core/Impala-Toolchain/gcc-4.9.2/include/c++/4.9.2/bits/stl_algo.h:1827
#15 0x02cbd01b in

[jira] [Created] (IMPALA-6483) Impala Doc: Describe the query option for query time limit

2018-02-06 Thread Alex Rodoni (JIRA) 
Alex Rodoni created IMPALA-6483:
---

 Summary: Impala Doc: Describe the query option for query time limit
 Key: IMPALA-6483
 URL: https://issues.apache.org/jira/browse/IMPALA-6483
 Project: IMPALA
  Issue Type: Sub-task
  Components: Docs
Reporter: Alex Rodoni
Assignee: Alex Rodoni






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IMPALA-6482) Add query option for query time limit

2018-02-06 Thread Tim Armstrong (JIRA) 
Tim Armstrong created IMPALA-6482:
-

 Summary: Add query option for query time limit
 Key: IMPALA-6482
 URL: https://issues.apache.org/jira/browse/IMPALA-6482
 Project: IMPALA
  Issue Type: Sub-task
  Components: Backend
Affects Versions: Impala 2.11.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


We should add a query option that cancels queries after a time limit. This can 
prevent runaway queries. This is different from the current timeout because it 
isn't reset.

I think the time limit should start after admission, so that it doesn't include 
time spent in the admission control queue.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)