[jira] [Commented] (HIVE-13237) Select parquet struct field with upper case throws NPE
[
https://issues.apache.org/jira/browse/HIVE-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190590#comment-15190590
]
Hive QA commented on HIVE-13237:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792166/HIVE-13237.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9804 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7216/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7216/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12792166 - PreCommit-HIVE-TRUNK-Build
> Select parquet struct field with upper case throws NPE
> --
>
> Key: HIVE-13237
> URL: https://issues.apache.org/jira/browse/HIVE-13237
> Project: Hive
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: HIVE-13237.1.patch
>
>
> Query "select msg.fieldone from test" throws NPE if msg's fieldone is
> actually fieldOne:
> {noformat}
> 2016-03-08 17:30:57,772 ERROR [main]: exec.FetchTask
> (FetchTask.java:initialize(86)) - java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:61)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:980)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:63)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13265) Query consists of union all and mapjoin, throw Exception “Unable to deserialize reduce input key”
[ https://issues.apache.org/jira/browse/HIVE-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ping Lu updated HIVE-13265: --- Attachment: explain2.txt execution2.txt execution1.txt explain1.txt > Query consists of union all and mapjoin, throw Exception “Unable to > deserialize reduce input key” > - > > Key: HIVE-13265 > URL: https://issues.apache.org/jira/browse/HIVE-13265 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.1 > Environment: Hadoop2.4.0 Hive0.13.1 >Reporter: Ping Lu > Attachments: execution1.txt, execution2.txt, explain1.txt, > explain2.txt > > > Steps to reproduce > Prepare: > create four test tables and load data > create table tmp_test1(col1 string); > create table tmp_test2(col1 string); > create table tmp_test3(col1 string,col2 string) row format delimited > fields terminated by "\t"; > create table tmp_test4(col1 string); > load data local inpath "test3" into table tmp_test1; // 6 rows > load data local inpath "test3" into table tmp_test2; // 5 rows > load data local inpath "test3" into table tmp_test3; // 6 rows > load data local inpath "test4" into table tmp_test4; // 311 rows, > 26670421Byte（>25M） > Query1: error encountered while executing > set hive.auto.convert.join=true; > select > sq.col1, > count(distinct sq.col2) num > from( > select > col1, > null col2 > from > tmp_test1 > union all > select > col1, > null col2 > from > tmp_test2 > union all > select > col1, > col2 > from > tmp_test3 > )sq --sq'size is far smaller than 25M > join > tmp_test4 ta > ON sq.col1 = ta.col1 > group by sq.col1; > when set hive.auto.convert.join to true, join was converted to MapJoin > and sq was chosen as the small table. > Query2: SELECT query got correct result > set hive.auto.convert.join=false; > select > sq.col1, > count(distinct sq.col2) num > from( > select > col1, > null col2 > from > tmp_test1 > union all > select > col1, > null col2 > from > tmp_test2 > union all > select > col1, > col2 > from > tmp_test3 > )sq > join > tmp_test4 ta > ON sq.col1 = ta.col1 > group by sq.col1; > the execute plan for Query1 names explain1.txt . > the hive execution logs for Query1: SELECT statement names execution1.txt . > the execute plan for the Query2 names explain2.txt . > the hive execution logs for Query2 names execution2.txt . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-4570: --- Attachment: HIVE-4570.06.patch > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.03.patch, HIVE-4570.04.patch, > HIVE-4570.04.patch, HIVE-4570.06.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190536#comment-15190536 ] Rajat Khandelwal commented on HIVE-4570: Seeing as the job has been waiting too long in the pre-commit queue, I'm merging with master and submitting patch again. > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.03.patch, HIVE-4570.04.patch, > HIVE-4570.04.patch, HIVE-4570.06.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4570) More information to user on GetOperationStatus in Hive Server2 when query is still executing
[ https://issues.apache.org/jira/browse/HIVE-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190535#comment-15190535 ] Rajat Khandelwal commented on HIVE-4570: Taking patch from reviewboard and attaching > More information to user on GetOperationStatus in Hive Server2 when query is > still executing > > > Key: HIVE-4570 > URL: https://issues.apache.org/jira/browse/HIVE-4570 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Amareshwari Sriramadasu >Assignee: Rajat Khandelwal > Attachments: HIVE-4570.01.patch, HIVE-4570.02.patch, > HIVE-4570.03.patch, HIVE-4570.03.patch, HIVE-4570.04.patch, > HIVE-4570.04.patch, HIVE-4570.06.patch > > > Currently in Hive Server2, when the query is still executing only the status > is set as STILL_EXECUTING. > This issue is to give more information to the user such as progress and > running job handles, if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13198) Authorization issues with cascading views
[ https://issues.apache.org/jira/browse/HIVE-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190504#comment-15190504 ] Ashutosh Chauhan commented on HIVE-13198: - +1 > Authorization issues with cascading views > - > > Key: HIVE-13198 > URL: https://issues.apache.org/jira/browse/HIVE-13198 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-13198.01.patch, HIVE-13198.02.patch > > > Here is a use case. They have a base table t1, from which they create a view > v1. They further create a view v2 from v1 by applying a filter. User has > access to only view v2, not view v1 or table t1. When user tries to access > v2, they are denied access. > Steps to recreate: > There is a base table t1 that exists in the default database with primary key > id and some employee data (name, ssn etc) > Create view v1 - “create view v1 as select * from default.t1;” > Created v2 - “create view v2 as select * from v1 where id =1;” > Permissions provided for user to select all columns from view v2. When user > runs select * from v2, hive throws an error “user does not have permissions > to select view v1". > Apparently Hive is converting the query to underlying views. > SELECT * FROM v2 LIMIT 100 > To > select `v1`.`id`, `v1`.`name`, `v1`.`ssn`, `v1`.`join_date`, `v1`.`location` > from `hr`.`v1` where `v1`.`id`=1 > Hive should only check for permissions for the view being run in the query, > not any parent views. (This is consistent with ORACLE). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13257) GroupBy with column alias does not support AVG
[
https://issues.apache.org/jira/browse/HIVE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190491#comment-15190491
]
Ashutosh Chauhan commented on HIVE-13257:
-
I have done some digging into this and following is what I have found:
Consider:
select col1, sum(col2) from t group by 1;
There are multiple ways on how above can be interpreted. When I read the
standard, AFAICT constant literal is not allowed in group by expression, so
this should throw an error. However, testing this on different DBs I found
different behavior.
Oracle & SQL server throws error for this. However, MySQL & Postgres assumes 1
in gby clause refers to position 1 in select list, so they translate this
effectively into:
select col1, sum(col2) from t group by col1;
Another interpretation for this is assume its group by on constant, meaning
there is one group for full table.
Hive of course has a config variable using which you can get either behavior
meaning different result set with different value of config variable.
Other variant for this is:
select sum(col2) from t group by 1;
MySQL & Postgres throws error for this consistent with their above
interpretation. SQL Server also throws error for this. However, Oracle
magically interprets this as group by on constant.
Yet another variant:
select 1 from t group by 1;
For this one MySQL, Postgres & Oracle can execute the query. SQL Server throws
error.
So, it seems only SQL Server is compliant with standard and other DBs do
different things depending on context.
I am wondering whether Hive should change its default behavior being consistent
with standard and start throwing exception whenever there it encounters a
constant literal in group by expression. Of course, their will be a config to
get back old behavior. Thoughts?
> GroupBy with column alias does not support AVG
> --
>
> Key: HIVE-13257
> URL: https://issues.apache.org/jira/browse/HIVE-13257
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>
> For the following query, with hive.groupby.orderby.position.alias set to true
> {code:title=Query}
> SELECT Avg(`t0`.`x_measure__0`) AS `avg_calculation_270497503505567749_ok`
> FROM (SELECT `store_sales`.`ss_ticket_number` AS `ss_ticket_number`,
>Sum(`store_sales`.`ss_net_paid`) AS `x_measure__0`
> FROM `store_sales` `store_sales`
>JOIN `item` `item`
> ON ( `store_sales`.`ss_item_sk` = `item`.`i_item_sk` )
> GROUP BY `store_sales`.`ss_ticket_number`) `t0`
> GROUP BY 1
> HAVING ( Count(1) > 0 );
> {code}
> it throws the following exception
> {code:title=Exception}
> FAILED: SemanticException [Error 10128]: Line 2:7 Not yet supported place for
> UDAF 'Avg’
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13257) GroupBy with column alias does not support AVG
[
https://issues.apache.org/jira/browse/HIVE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190488#comment-15190488
]
Ashutosh Chauhan commented on HIVE-13257:
-
If you set {{hive.groupby.orderby.position.alias}} to true then query doesnt
make sense. It effectively mean do GBY on avg column. So, throwing exception is
indeed correct here.
> GroupBy with column alias does not support AVG
> --
>
> Key: HIVE-13257
> URL: https://issues.apache.org/jira/browse/HIVE-13257
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>
> For the following query, with hive.groupby.orderby.position.alias set to true
> {code:title=Query}
> SELECT Avg(`t0`.`x_measure__0`) AS `avg_calculation_270497503505567749_ok`
> FROM (SELECT `store_sales`.`ss_ticket_number` AS `ss_ticket_number`,
>Sum(`store_sales`.`ss_net_paid`) AS `x_measure__0`
> FROM `store_sales` `store_sales`
>JOIN `item` `item`
> ON ( `store_sales`.`ss_item_sk` = `item`.`i_item_sk` )
> GROUP BY `store_sales`.`ss_ticket_number`) `t0`
> GROUP BY 1
> HAVING ( Count(1) > 0 );
> {code}
> it throws the following exception
> {code:title=Exception}
> FAILED: SemanticException [Error 10128]: Line 2:7 Not yet supported place for
> UDAF 'Avg’
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12977) Pass credentials in the current UGI while creating Tez session
[
https://issues.apache.org/jira/browse/HIVE-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-12977:
-
Status: Patch Available (was: Open)
> Pass credentials in the current UGI while creating Tez session
> --
>
> Key: HIVE-12977
> URL: https://issues.apache.org/jira/browse/HIVE-12977
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Reporter: Vinoth Sathappan
>Assignee: Vinoth Sathappan
> Attachments: HIVE-12977.1.patch
>
>
> The credentials present in the current UGI i.e.
> UserGroupInformation.getCurrentUser().getCredentials() isn't passed to the
> Tez session. It is instantiated with null credentials
> session = TezClient.create("HIVE-" + sessionId, tezConfig, true,
> commonLocalResources, null);
> In this case, Tez fails to access resources even if the tokens are available
> in memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13236) LLAP: token renewal interval needs to be set
[ https://issues.apache.org/jira/browse/HIVE-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13236: Resolution: Fixed Fix Version/s: 2.0.1 2.1.0 Status: Resolved (was: Patch Available) Committed to branches. Thanks for the review! > LLAP: token renewal interval needs to be set > > > Key: HIVE-13236 > URL: https://issues.apache.org/jira/browse/HIVE-13236 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0, 2.0.1 > > Attachments: HIVE-13236.01.patch, HIVE-13236.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[ https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12558: - Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master > LLAP: output QueryFragmentCounters somewhere > > > Key: HIVE-12558 > URL: https://issues.apache.org/jira/browse/HIVE-12558 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch, > HIVE-12558.3.patch, HIVE-12558.3.patch, HIVE-12558.4.patch, > HIVE-12558.wip.patch, sample-output.png, sample-output2.png > > > Right now, LLAP logs counters for every fragment; most of them are IO related > and could be very useful, they also include table names so that things like > cache hit ratio, etc., could be calculated for every table. > We need to output them to some metrics system (preserving the breakdown by > table, possibly also adding query ID or even stage) so that they'd be usable > without grep/sed/awk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-9660: --- Attachment: HIVE-9660.WIP0.patch WIP patch that takes care of the reading; the writing is only done for compressed path and not done for string writer yet cause its logic is different... whether it works at all is an open question. Also, my head hurts now... I feel like after researching how Kerberos works. > store end offset of compressed data for RG in RowIndex in ORC > - > > Key: HIVE-9660 > URL: https://issues.apache.org/jira/browse/HIVE-9660 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-9660.WIP0.patch > > > Right now the end offset is estimated, which in some cases results in tons of > extra data being read. > We can add a separate array to RowIndex (positions_v2?) that stores number of > compressed buffers for each RG, or end offset, or something, to remove this > estimation magic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4662) first_value can't have more than one order by column
[
https://issues.apache.org/jira/browse/HIVE-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190383#comment-15190383
]
Ashutosh Chauhan commented on HIVE-4662:
+1 Patch looks good.
Can you also add following test:
{code}
select last_value(i) over (partition by si, bo order by i, f desc range
current row) from over10k limit 100;
select row_number() over (partition by si, bo order by i, f desc range
unbounded preceding) from over10k limit 100;
{code}
> first_value can't have more than one order by column
>
>
> Key: HIVE-4662
> URL: https://issues.apache.org/jira/browse/HIVE-4662
> Project: Hive
> Issue Type: Bug
> Components: SQL
>Affects Versions: 0.11.0
>Reporter: Frans Drijver
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-4662.01.patch, HIVE-4662.patch
>
>
> In the current implementation of the first_value function, it's not allowed
> to have more than one (1) order by column, as so:
> {quote}
> select distinct
> first_value(kastr.DEWNKNR) over ( partition by kastr.DEKTRNR order by
> kastr.DETRADT, kastr.DEVPDNR )
> from RTAVP_DRKASTR kastr
> ;
> {quote}
> Error given:
> {quote}
> FAILED: SemanticException Range based Window Frame can have only 1 Sort Key
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13263) Vectorization: Unable to vectorize regexp_extract " Udf: GenericUDFBridge, is not supported"
[ https://issues.apache.org/jira/browse/HIVE-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13263: Status: Patch Available (was: Open) > Vectorization: Unable to vectorize regexp_extract " Udf: GenericUDFBridge, is > not supported" > > > Key: HIVE-13263 > URL: https://issues.apache.org/jira/browse/HIVE-13263 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13263.01.patch > > > Add regexp_extract to the UDFs we bridge to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13263) Vectorization: Unable to vectorize regexp_extract " Udf: GenericUDFBridge, is not supported"
[ https://issues.apache.org/jira/browse/HIVE-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13263: Attachment: HIVE-13263.01.patch > Vectorization: Unable to vectorize regexp_extract " Udf: GenericUDFBridge, is > not supported" > > > Key: HIVE-13263 > URL: https://issues.apache.org/jira/browse/HIVE-13263 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13263.01.patch > > > Add regexp_extract to the UDFs we bridge to. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13242) DISTINCT keyword is dropped by the parser for windowing
[
https://issues.apache.org/jira/browse/HIVE-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190350#comment-15190350
]
Ashutosh Chauhan commented on HIVE-13242:
-
Did you forgot to attach a patch : )
> DISTINCT keyword is dropped by the parser for windowing
> ---
>
> Key: HIVE-13242
> URL: https://issues.apache.org/jira/browse/HIVE-13242
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> To reproduce, the following query can be used:
> {noformat}
> select distinct first_value(t) over ( partition by si order by i, b ) from
> over10k limit 100;
> {noformat}
> The distinct keyword is ignored and duplicates are produced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12558) LLAP: output QueryFragmentCounters somewhere
[
https://issues.apache.org/jira/browse/HIVE-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190282#comment-15190282
]
Hive QA commented on HIVE-12558:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792118/HIVE-12558.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9803 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7214/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7214/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7214/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12792118 - PreCommit-HIVE-TRUNK-Build
> LLAP: output QueryFragmentCounters somewhere
>
>
> Key: HIVE-12558
> URL: https://issues.apache.org/jira/browse/HIVE-12558
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12558.1.patch, HIVE-12558.2.patch,
> HIVE-12558.3.patch, HIVE-12558.3.patch, HIVE-12558.4.patch,
> HIVE-12558.wip.patch, sample-output.png, sample-output2.png
>
>
> Right now, LLAP logs counters for every fragment; most of them are IO related
> and could be very useful, they also include table names so that things like
> cache hit ratio, etc., could be calculated for every table.
> We need to output them to some metrics system (preserving the breakdown by
> table, possibly also adding query ID or even stage) so that they'd be usable
> without grep/sed/awk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13262) LLAP: Remove log levels from DebugUtils
[ https://issues.apache.org/jira/browse/HIVE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190279#comment-15190279 ] Sergey Shelukhin commented on HIVE-13262: - Loggers should be organized logically, not by class, similar to current boolean checks > LLAP: Remove log levels from DebugUtils > --- > > Key: HIVE-13262 > URL: https://issues.apache.org/jira/browse/HIVE-13262 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > > DebugUtils has many hardcoded log levels. To enable logging we need to > recompile code with desired value. Instead configure add loggers for these > classes with log levels via log4j properties. Also use parametrized logging > in IO elevator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-10632: - Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, > HIVE-10632.6.patch, HIVE-10632.7.patch, HIVE-10632.branch-1.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190261#comment-15190261 ] Wei Zheng commented on HIVE-10632: -- Committed to master and branch-1 > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, > HIVE-10632.6.patch, HIVE-10632.7.patch, HIVE-10632.branch-1.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-10632: - Attachment: HIVE-10632.branch-1.patch > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, > HIVE-10632.6.patch, HIVE-10632.7.patch, HIVE-10632.branch-1.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12832) RDBMS schema changes for HIVE-11388
[ https://issues.apache.org/jira/browse/HIVE-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-12832: -- Component/s: Transactions > RDBMS schema changes for HIVE-11388 > --- > > Key: HIVE-12832 > URL: https://issues.apache.org/jira/browse/HIVE-12832 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12382.patch, HIVE-12832.3.patch, > HIVE-12832.uber.2.branch-1.patch, HIVE-12832.uber.2.branch-2.0.patch, > HIVE-12832.uber.2.patch, HIVE-12832.uber.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13260) ReduceSinkDeDuplication throws exception when pRS key is empty
[
https://issues.apache.org/jira/browse/HIVE-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13260:
---
Status: Patch Available (was: Open)
> ReduceSinkDeDuplication throws exception when pRS key is empty
> --
>
> Key: HIVE-13260
> URL: https://issues.apache.org/jira/browse/HIVE-13260
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13260.01.patch, HIVE-13260.02.patch
>
>
> Steps to reproduce:
> {code}
> set hive.mapred.mode=nonstrict;
> set hive.cbo.enable=false;
> set hive.map.aggr=false;
> set hive.groupby.skewindata=false;
> set mapred.reduce.tasks=31;
> select
> compute_stats(a,16),compute_stats(b,16),compute_stats(c,16),compute_stats(d,16)
> from
> (
> select
> avg(DISTINCT substr(src.value,5)) as a,
> max(substr(src.value,5)) as b,
> variance(substr(src.value,5)) as c,
> var_samp(substr(src.value,5)) as d
> from src)subq;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13260) ReduceSinkDeDuplication throws exception when pRS key is empty
[
https://issues.apache.org/jira/browse/HIVE-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13260:
---
Status: Open (was: Patch Available)
> ReduceSinkDeDuplication throws exception when pRS key is empty
> --
>
> Key: HIVE-13260
> URL: https://issues.apache.org/jira/browse/HIVE-13260
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13260.01.patch, HIVE-13260.02.patch
>
>
> Steps to reproduce:
> {code}
> set hive.mapred.mode=nonstrict;
> set hive.cbo.enable=false;
> set hive.map.aggr=false;
> set hive.groupby.skewindata=false;
> set mapred.reduce.tasks=31;
> select
> compute_stats(a,16),compute_stats(b,16),compute_stats(c,16),compute_stats(d,16)
> from
> (
> select
> avg(DISTINCT substr(src.value,5)) as a,
> max(substr(src.value,5)) as b,
> variance(substr(src.value,5)) as c,
> var_samp(substr(src.value,5)) as d
> from src)subq;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13261) Can not compute column stats for partition when schema evolves
[
https://issues.apache.org/jira/browse/HIVE-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13261:
---
Description:
To repro
{code}
CREATE TABLE partitioned1(a INT, b STRING) PARTITIONED BY(part INT) STORED AS
TEXTFILE;
insert into table partitioned1 partition(part=1) values(1, 'original'),(2,
'original'), (3, 'original'),(4, 'original');
-- Table-Non-Cascade ADD COLUMNS ...
alter table partitioned1 add columns(c int, d string);
insert into table partitioned1 partition(part=2) values(1, 'new', 10,
'ten'),(2, 'new', 20, 'twenty'), (3, 'new', 30, 'thirty'),(4, 'new', 40,
'forty');
insert into table partitioned1 partition(part=1) values(5, 'new', 100,
'hundred'),(6, 'new', 200, 'two hundred');
analyze table partitioned1 compute statistics for columns;
{code}
Error msg:
{code}
2016-03-10T14:55:43,205 ERROR [abc3eb8d-7432-47ae-b76f-54c8d7020312 main[]]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(177)) -
NoSuchObjectException(message:Column c for which stats gathering is requested
doesn't exist.)
at
org.apache.hadoop.hive.metastore.ObjectStore.writeMPartitionColumnStatistics(ObjectStore.java:6492)
at
org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:6574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
{code}
was:
To repro
{code}
CREATE TABLE partitioned1(a INT, b STRING) PARTITIONED BY(part INT) STORED AS
TEXTFILE;
insert into table partitioned1 partition(part=1) values(1, 'original'),(2,
'original'), (3, 'original'),(4, 'original');
-- Table-Non-Cascade ADD COLUMNS ...
alter table partitioned1 add columns(c int, d string);
insert into table partitioned1 partition(part=2) values(1, 'new', 10,
'ten'),(2, 'new', 20, 'twenty'), (3, 'new', 30, 'thirty'),(4, 'new', 40,
'forty');
insert into table partitioned1 partition(part=1) values(5, 'new', 100,
'hundred'),(6, 'new', 200, 'two hundred');
analyze table partitioned1 compute statistics for columns;
{code}
> Can not compute column stats for partition when schema evolves
> --
>
> Key: HIVE-13261
> URL: https://issues.apache.org/jira/browse/HIVE-13261
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> To repro
> {code}
> CREATE TABLE partitioned1(a INT, b STRING) PARTITIONED BY(part INT) STORED AS
> TEXTFILE;
> insert into table partitioned1 partition(part=1) values(1, 'original'),(2,
> 'original'), (3, 'original'),(4, 'original');
> -- Table-Non-Cascade ADD COLUMNS ...
> alter table partitioned1 add columns(c int, d string);
> insert into table partitioned1 partition(part=2) values(1, 'new', 10,
> 'ten'),(2, 'new', 20, 'twenty'), (3, 'new', 30, 'thirty'),(4, 'new', 40,
> 'forty');
> insert into table partitioned1 partition(part=1) values(5, 'new', 100,
> 'hundred'),(6, 'new', 200, 'two hundred');
> analyze table partitioned1 compute statistics for columns;
> {code}
> Error msg:
> {code}
> 2016-03-10T14:55:43,205 ERROR [abc3eb8d-7432-47ae-b76f-54c8d7020312 main[]]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(177)) -
> NoSuchObjectException(message:Column c for which stats gathering is requested
> doesn't exist.)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.writeMPartitionColumnStatistics(ObjectStore.java:6492)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:6574)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13260) ReduceSinkDeDuplication throws exception when pRS key is empty
[
https://issues.apache.org/jira/browse/HIVE-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190128#comment-15190128
]
Prasanth Jayachandran commented on HIVE-13260:
--
Can you add explain to the test case with RS dedup enable and disabled?
Also in the patch, you are copying the number of distribution keys from cRS but
not the fields. I suspect that. We might need a test run to see if that breaks
anything.
> ReduceSinkDeDuplication throws exception when pRS key is empty
> --
>
> Key: HIVE-13260
> URL: https://issues.apache.org/jira/browse/HIVE-13260
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13260.01.patch
>
>
> Steps to reproduce:
> {code}
> set hive.mapred.mode=nonstrict;
> set hive.cbo.enable=false;
> set hive.map.aggr=false;
> set hive.groupby.skewindata=false;
> set mapred.reduce.tasks=31;
> select
> compute_stats(a,16),compute_stats(b,16),compute_stats(c,16),compute_stats(d,16)
> from
> (
> select
> avg(DISTINCT substr(src.value,5)) as a,
> max(substr(src.value,5)) as b,
> variance(substr(src.value,5)) as c,
> var_samp(substr(src.value,5)) as d
> from src)subq;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[ https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-13243: --- Attachment: HIVE-13243.2.patch Uploaded a new patch with the fix to a typo in table name. > Hive drop table on encyption zone fails for external tables > --- > > Key: HIVE-13243 > URL: https://issues.apache.org/jira/browse/HIVE-13243 > Project: Hive > Issue Type: Bug > Components: Encryption, Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13243.1.patch, HIVE-13243.2.patch, HIVE-13243.patch > > > When dropping an external table with its data located in an encryption zone, > hive should not throw out MetaException(message:Unable to drop table because > it is in an encryption zone and trash is enabled. Use PURGE option to skip > trash.) in checkTrashPurgeCombination since the data should not get deleted > (or trashed) anyway regardless HDFS Trash is enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13185) orc.ReaderImp.ensureOrcFooter() method fails on small text files with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190072#comment-15190072
]
Illya Yalovyy commented on HIVE-13185:
--
Test link [1] shows only 2 failed tests. Both of them have been failing for 70+
builds.
1:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7206/testReport
> orc.ReaderImp.ensureOrcFooter() method fails on small text files with
> IndexOutOfBoundsException
> ---
>
> Key: HIVE-13185
> URL: https://issues.apache.org/jira/browse/HIVE-13185
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: HIVE-13185.1.patch
>
>
> Steps to reproduce:
> 1. Create a Text source table with one line of data:
> {code}
> create table src (id int);
> insert overwrite table src values (1);
> {code}
> 2. Create a target table:
> {code}
> create table trg (id int);
> {code}
> 3. Try to load small text file to the target table:
> {code}
> load data inpath 'user/hive/warehouse/src/00_0' into table trg;
> {code}
> *Error message:*
> {quote}
> FAILED: SemanticException Unable to load data to destination table. Error:
> java.lang.IndexOutOfBoundsException
> {quote}
> *Stack trace:*
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Unable to load data to
> destination table. Error: java.lang.IndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.ensureFileFormatsMatch(LoadSemanticAnalyzer.java:340)
> at
> org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:224)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:242)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:481)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:317)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1190)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1104)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13175: - Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch, > HIVE-13175.3.patch, HIVE-13175.4.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13206) Create a test-sources.jar when -Psources profile is invoked
[
https://issues.apache.org/jira/browse/HIVE-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190066#comment-15190066
]
Hive QA commented on HIVE-13206:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792115/HIVE-13206.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9801 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7213/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7213/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7213/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12792115 - PreCommit-HIVE-TRUNK-Build
> Create a test-sources.jar when -Psources profile is invoked
> ---
>
> Key: HIVE-13206
> URL: https://issues.apache.org/jira/browse/HIVE-13206
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-13206.1.patch, HIVE-13206.2.patch,
> HIVE-13206.3.patch
>
>
> It'd be nice to attach a test-sources jar alongside the others as part of the
> build, to provide test resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13175) Disallow making external tables transactional
[ https://issues.apache.org/jira/browse/HIVE-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190063#comment-15190063 ] Wei Zheng commented on HIVE-13175: -- Committed to master and branch-1 > Disallow making external tables transactional > - > > Key: HIVE-13175 > URL: https://issues.apache.org/jira/browse/HIVE-13175 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13175.1.patch, HIVE-13175.2.patch, > HIVE-13175.3.patch, HIVE-13175.4.patch > > > The fact that compactor rewrites contents of ACID tables is in conflict with > what is expected of external tables. > Conversely, end user can write to External table which certainly not what is > expected of ACID table. > So we should explicitly disallow making an external table ACID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[ https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190051#comment-15190051 ] Chaoyu Tang commented on HIVE-13243: Thanks [~spena]. A good catch. Yes, I meant encrypted_table_dp. I will create another patch for it. > Hive drop table on encyption zone fails for external tables > --- > > Key: HIVE-13243 > URL: https://issues.apache.org/jira/browse/HIVE-13243 > Project: Hive > Issue Type: Bug > Components: Encryption, Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13243.1.patch, HIVE-13243.patch > > > When dropping an external table with its data located in an encryption zone, > hive should not throw out MetaException(message:Unable to drop table because > it is in an encryption zone and trash is enabled. Use PURGE option to skip > trash.) in checkTrashPurgeCombination since the data should not get deleted > (or trashed) anyway regardless HDFS Trash is enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12619) Switching the field order within an array of structs causes the query to fail
[ https://issues.apache.org/jira/browse/HIVE-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-12619: --- Attachment: HIVE-12619.3.patch I attached a patch (v3) that is a little simpler. It has the qtests from Mohammad's patch (v2). [~spena], could you take a look? > Switching the field order within an array of structs causes the query to fail > - > > Key: HIVE-12619 > URL: https://issues.apache.org/jira/browse/HIVE-12619 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Ang Zhang >Assignee: Mohammad Kamrul Islam >Priority: Minor > Attachments: HIVE-12619.2.patch, HIVE-12619.3.patch > > > Switching the field order within an array of structs causes the query to fail > or return the wrong data for the fields, but switching the field order within > just a struct works. > How to reproduce: > Case1 if the two fields have the same type, query will return wrong data for > the fields > drop table if exists schema_test; > create table schema_test (msg array) stored > as parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 'abc2')), array(named_struct('f1', 'efg', 'f2', 'efg2'))) from one > limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":"efg2"}] > --[{"f1":"abc","f2":"abc2"}] > alter table schema_test change msg msg array ; > select * from schema_test; > --returns > --[{"f2":"efg","f1":"efg2"}] > --[{"f2":"abc","f1":"abc2"}] > Case2: if the two fields have different type, the query will fail > drop table if exists schema_test; > create table schema_test (msg array ) stored as > parquet; > insert into table schema_test select stack(2, array(named_struct('f1', 'abc', > 'f2', 1)), array(named_struct('f1', 'efg', 'f2', 2))) from one limit 2; > select * from schema_test; > --returns > --[{"f1":"efg","f2":2}] > --[{"f1":"abc","f2":1}] > alter table schema_test change msg msg array ; > select * from schema_test; > Failed with exception > java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to > org.apache.hadoop.io.IntWritable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12481) Occasionally "Request is a replay" will be thrown from HS2
[
https://issues.apache.org/jira/browse/HIVE-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190037#comment-15190037
]
Aihua Xu commented on HIVE-12481:
-
JDBC connection string doesn't restrict that. We can add the ones which make
sense.
> Occasionally "Request is a replay" will be thrown from HS2
> --
>
> Key: HIVE-12481
> URL: https://issues.apache.org/jira/browse/HIVE-12481
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12481.2.patch, HIVE-12481.3.patch, HIVE-12481.patch
>
>
> We have seen the following exception thrown from HS2 in secured cluster when
> many queries are running simultaneously on single HS2 instance.
> The cause I can guess is that it happens that two queries are submitted at
> the same time and have the same timestamp. For such case, we can add a retry
> for the query.
>
> {noformat}
> 2015-11-18 16:12:33,117 ERROR org.apache.thrift.transport.TSaslTransport:
> SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 14 more
> Caused by: KrbException: Request is a replay (34)
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
> at sun.security.krb5.KrbApReq.(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:108)
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[
https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190034#comment-15190034
]
Sergio Peña commented on HIVE-13243:
In {{encryption_drop_partition.q}}, I see you're dropping the external table
with purge:
{noformat}
+DROP TABLE encrypted_ext_table_dp;
+
+SELECT * FROM encrypted_table_dp;
ALTER TABLE encrypted_table_dp DROP PARTITION (p='2014-09-23');
SELECT * FROM encrypted_table_dp;
ALTER TABLE encrypted_table_dp DROP PARTITION (p='2014-09-23') PURGE;
SELECT * FROM encrypted_table_dp;
+DROP TABLE encrypted_ext_table_dp PURGE;
{noformat}
Didn't you mean to drop {{encrypted_table_dp}} instead?
> Hive drop table on encyption zone fails for external tables
> ---
>
> Key: HIVE-13243
> URL: https://issues.apache.org/jira/browse/HIVE-13243
> Project: Hive
> Issue Type: Bug
> Components: Encryption, Metastore
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13243.1.patch, HIVE-13243.patch
>
>
> When dropping an external table with its data located in an encryption zone,
> hive should not throw out MetaException(message:Unable to drop table because
> it is in an encryption zone and trash is enabled. Use PURGE option to skip
> trash.) in checkTrashPurgeCombination since the data should not get deleted
> (or trashed) anyway regardless HDFS Trash is enabled or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12481) Occasionally "Request is a replay" will be thrown from HS2
[
https://issues.apache.org/jira/browse/HIVE-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190027#comment-15190027
]
Yongzhi Chen commented on HIVE-12481:
-
The change looks fine. One question related to the newly added param retries,
do we need follow some specs when add session params in jdbc connection string,
or any param can be added?
> Occasionally "Request is a replay" will be thrown from HS2
> --
>
> Key: HIVE-12481
> URL: https://issues.apache.org/jira/browse/HIVE-12481
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12481.2.patch, HIVE-12481.3.patch, HIVE-12481.patch
>
>
> We have seen the following exception thrown from HS2 in secured cluster when
> many queries are running simultaneously on single HS2 instance.
> The cause I can guess is that it happens that two queries are submitted at
> the same time and have the same timestamp. For such case, we can add a retry
> for the query.
>
> {noformat}
> 2015-11-18 16:12:33,117 ERROR org.apache.thrift.transport.TSaslTransport:
> SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 14 more
> Caused by: KrbException: Request is a replay (34)
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
> at sun.security.krb5.KrbApReq.(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:108)
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189985#comment-15189985 ] Jarek Jarcec Cecho commented on HIVE-13251: --- I'm no longer involved very actively in the project [~aihuaxu], so whatever the community decides :) > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro definition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[ https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-13243: --- Attachment: HIVE-13243.1.patch Add tests for drop partitions of an external table. > Hive drop table on encyption zone fails for external tables > --- > > Key: HIVE-13243 > URL: https://issues.apache.org/jira/browse/HIVE-13243 > Project: Hive > Issue Type: Bug > Components: Encryption, Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13243.1.patch, HIVE-13243.patch > > > When dropping an external table with its data located in an encryption zone, > hive should not throw out MetaException(message:Unable to drop table because > it is in an encryption zone and trash is enabled. Use PURGE option to skip > trash.) in checkTrashPurgeCombination since the data should not get deleted > (or trashed) anyway regardless HDFS Trash is enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13242) DISTINCT keyword is dropped by the parser for windowing
[
https://issues.apache.org/jira/browse/HIVE-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13242:
---
Status: Patch Available (was: In Progress)
> DISTINCT keyword is dropped by the parser for windowing
> ---
>
> Key: HIVE-13242
> URL: https://issues.apache.org/jira/browse/HIVE-13242
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> To reproduce, the following query can be used:
> {noformat}
> select distinct first_value(t) over ( partition by si order by i, b ) from
> over10k limit 100;
> {noformat}
> The distinct keyword is ignored and duplicates are produced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13242) DISTINCT keyword is dropped by the parser for windowing
[
https://issues.apache.org/jira/browse/HIVE-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13242 started by Jesus Camacho Rodriguez.
--
> DISTINCT keyword is dropped by the parser for windowing
> ---
>
> Key: HIVE-13242
> URL: https://issues.apache.org/jira/browse/HIVE-13242
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> To reproduce, the following query can be used:
> {noformat}
> select distinct first_value(t) over ( partition by si order by i, b ) from
> over10k limit 100;
> {noformat}
> The distinct keyword is ignored and duplicates are produced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12721) Add UUID built in function
[ https://issues.apache.org/jira/browse/HIVE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189956#comment-15189956 ] Aihua Xu commented on HIVE-12721: - You can't have such unit test. Maybe it's good to verify the length of the code = 36 and the values regenerated every time will be different. > Add UUID built in function > -- > > Key: HIVE-12721 > URL: https://issues.apache.org/jira/browse/HIVE-12721 > Project: Hive > Issue Type: Improvement > Components: UDF >Reporter: Jeremy Beard >Assignee: Jeremy Beard > Attachments: HIVE-12721.patch > > > A UUID function would be very useful for ETL jobs that need to generate > surrogate keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13260) ReduceSinkDeDuplication throws exception when pRS key is empty
[
https://issues.apache.org/jira/browse/HIVE-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13260:
---
Status: Patch Available (was: Open)
> ReduceSinkDeDuplication throws exception when pRS key is empty
> --
>
> Key: HIVE-13260
> URL: https://issues.apache.org/jira/browse/HIVE-13260
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13260.01.patch
>
>
> Steps to reproduce:
> {code}
> set hive.mapred.mode=nonstrict;
> set hive.cbo.enable=false;
> set hive.map.aggr=false;
> set hive.groupby.skewindata=false;
> set mapred.reduce.tasks=31;
> select
> compute_stats(a,16),compute_stats(b,16),compute_stats(c,16),compute_stats(d,16)
> from
> (
> select
> avg(DISTINCT substr(src.value,5)) as a,
> max(substr(src.value,5)) as b,
> variance(substr(src.value,5)) as c,
> var_samp(substr(src.value,5)) as d
> from src)subq;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13260) ReduceSinkDeDuplication throws exception when pRS key is empty
[
https://issues.apache.org/jira/browse/HIVE-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13260:
---
Attachment: HIVE-13260.01.patch
[~prasanth_j], could u please review it? It is related to the issue that I
mentioned to you.
> ReduceSinkDeDuplication throws exception when pRS key is empty
> --
>
> Key: HIVE-13260
> URL: https://issues.apache.org/jira/browse/HIVE-13260
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13260.01.patch
>
>
> Steps to reproduce:
> {code}
> set hive.mapred.mode=nonstrict;
> set hive.cbo.enable=false;
> set hive.map.aggr=false;
> set hive.groupby.skewindata=false;
> set mapred.reduce.tasks=31;
> select
> compute_stats(a,16),compute_stats(b,16),compute_stats(c,16),compute_stats(d,16)
> from
> (
> select
> avg(DISTINCT substr(src.value,5)) as a,
> max(substr(src.value,5)) as b,
> variance(substr(src.value,5)) as c,
> var_samp(substr(src.value,5)) as d
> from src)subq;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13260) ReduceSinkDeDuplication throws exception when pRS key is empty
[
https://issues.apache.org/jira/browse/HIVE-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13260:
---
Issue Type: Bug (was: Sub-task)
Parent: (was: HIVE-11160)
> ReduceSinkDeDuplication throws exception when pRS key is empty
> --
>
> Key: HIVE-13260
> URL: https://issues.apache.org/jira/browse/HIVE-13260
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> Steps to reproduce:
> {code}
> set hive.mapred.mode=nonstrict;
> set hive.cbo.enable=false;
> set hive.map.aggr=false;
> set hive.groupby.skewindata=false;
> set mapred.reduce.tasks=31;
> select
> compute_stats(a,16),compute_stats(b,16),compute_stats(c,16),compute_stats(d,16)
> from
> (
> select
> avg(DISTINCT substr(src.value,5)) as a,
> max(substr(src.value,5)) as b,
> variance(substr(src.value,5)) as c,
> var_samp(substr(src.value,5)) as d
> from src)subq;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13231) Show helpful error message on failure to create table in nested directory
[
https://issues.apache.org/jira/browse/HIVE-13231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuben Kuhnert updated HIVE-13231:
--
Status: Patch Available (was: Open)
> Show helpful error message on failure to create table in nested directory
> -
>
> Key: HIVE-13231
> URL: https://issues.apache.org/jira/browse/HIVE-13231
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
>Priority: Minor
> Attachments: HIVE-13231.01.patch, HIVE-13231.02.patch
>
>
> cannot store data in a directory whose parent doesn't exist, even though the
> target dir does have an existing ancestor on HDFS. This occurs when trying to
> perform {{create table }}.
> {code}
> 0: jdbc:hive2://10.17.81.192:1/default> create table test3 location
> '/user/hive/data/yshi/nonexisting/test3' as select * from sample_07;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> Error message:
> 2015-10-29 19:04:46,323 ERROR org.apache.hadoop.hive.ql.exec.Task: Failed
> with exception Unable to rename:
> hdfs://host-10-17-81-192.coe.cloudera.com:8020/user/hive/warehouse/.hive-staging_hive_2015-10-29_19-04-08_375_5385987873542863570-3/-ext-10001
> to: /user/hive/data/yshi/nonexisting/test3
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename:
> hdfs://host-10-17-81-192.coe.cloudera.com:8020/user/hive/warehouse/.hive-staging_hive_2015-10-29_19-04-08_375_5385987873542863570-3/-ext-10001
> to: /user/hive/data/yshi/nonexisting/test3
> at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:101)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:209)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:144)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:212)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189918#comment-15189918 ] Szehon Ho commented on HIVE-13251: -- It makes sense to me to support for those users for the previous version at little cost, +1 > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro definition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12481) Occasionally "Request is a replay" will be thrown from HS2
[
https://issues.apache.org/jira/browse/HIVE-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189911#comment-15189911
]
Aihua Xu commented on HIVE-12481:
-
[~ychena], [~ctang.ma] and [~szehon] Can you take a look at the change? This is
to retry from the JDBC client side if there is any connection and this
authentication issue.
> Occasionally "Request is a replay" will be thrown from HS2
> --
>
> Key: HIVE-12481
> URL: https://issues.apache.org/jira/browse/HIVE-12481
> Project: Hive
> Issue Type: Improvement
> Components: Authentication
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12481.2.patch, HIVE-12481.3.patch, HIVE-12481.patch
>
>
> We have seen the following exception thrown from HS2 in secured cluster when
> many queries are running simultaneously on single HS2 instance.
> The cause I can guess is that it happens that two queries are submitted at
> the same time and have the same timestamp. For such case, we can add a retry
> for the query.
>
> {noformat}
> 2015-11-18 16:12:33,117 ERROR org.apache.thrift.transport.TSaslTransport:
> SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request
> is a replay (34))]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
> at
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
> at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:739)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:736)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:736)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism
> level: Request is a replay (34))
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
> at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
> ... 14 more
> Caused by: KrbException: Request is a replay (34)
> at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:308)
> at sun.security.krb5.KrbApReq.(KrbApReq.java:144)
> at
> sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:108)
> at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13235) Insert from select generates incorrect result when hive.optimize.constant.propagation is on
[
https://issues.apache.org/jira/browse/HIVE-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13235:
Status: Patch Available (was: Open)
Attached the patch-1: seems we are trying to resolve the column info from the
parent OP to child OP by alias. But alias should be visible to the children and
the internalName is visible to the parents.
> Insert from select generates incorrect result when
> hive.optimize.constant.propagation is on
> ---
>
> Key: HIVE-13235
> URL: https://issues.apache.org/jira/browse/HIVE-13235
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13235.1.patch
>
>
> The following query returns incorrect result when constant optimization is
> turned on. The subquery happens to have an alias p1 to be the same as the
> input partition name. Constant optimizer will optimize it incorrectly as the
> constant.
> When constant optimizer is turned off, we will get the correct result.
> {noformat}
> set hive.cbo.enable=false;
> set hive.optimize.constant.propagation = true;
> create table t1(c1 string, c2 double) partitioned by (p1 string, p2 string);
> create table t2(p1 double, c2 string);
> insert into table t1 partition(p1='40', p2='p2') values('c1', 0.0);
> INSERT OVERWRITE TABLE t2 select if((c2 = 0.0), c2, '0') as p1, 2 as p2 from
> t1 where c1 = 'c1' and p1 = '40';
> select * from t2;
> 40 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13235) Insert from select generates incorrect result when hive.optimize.constant.propagation is on
[
https://issues.apache.org/jira/browse/HIVE-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-13235:
Attachment: HIVE-13235.1.patch
> Insert from select generates incorrect result when
> hive.optimize.constant.propagation is on
> ---
>
> Key: HIVE-13235
> URL: https://issues.apache.org/jira/browse/HIVE-13235
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13235.1.patch
>
>
> The following query returns incorrect result when constant optimization is
> turned on. The subquery happens to have an alias p1 to be the same as the
> input partition name. Constant optimizer will optimize it incorrectly as the
> constant.
> When constant optimizer is turned off, we will get the correct result.
> {noformat}
> set hive.cbo.enable=false;
> set hive.optimize.constant.propagation = true;
> create table t1(c1 string, c2 double) partitioned by (p1 string, p2 string);
> create table t2(p1 double, c2 string);
> insert into table t1 partition(p1='40', p2='p2') values('c1', 0.0);
> INSERT OVERWRITE TABLE t2 select if((c2 = 0.0), c2, '0') as p1, 2 as p2 from
> t1 where c1 = 'c1' and p1 = '40';
> select * from t2;
> 40 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13231) Show helpful error message on failure to create table in nested directory
[
https://issues.apache.org/jira/browse/HIVE-13231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuben Kuhnert updated HIVE-13231:
--
Status: Open (was: Patch Available)
> Show helpful error message on failure to create table in nested directory
> -
>
> Key: HIVE-13231
> URL: https://issues.apache.org/jira/browse/HIVE-13231
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
>Priority: Minor
> Attachments: HIVE-13231.01.patch, HIVE-13231.02.patch
>
>
> cannot store data in a directory whose parent doesn't exist, even though the
> target dir does have an existing ancestor on HDFS. This occurs when trying to
> perform {{create table }}.
> {code}
> 0: jdbc:hive2://10.17.81.192:1/default> create table test3 location
> '/user/hive/data/yshi/nonexisting/test3' as select * from sample_07;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> Error message:
> 2015-10-29 19:04:46,323 ERROR org.apache.hadoop.hive.ql.exec.Task: Failed
> with exception Unable to rename:
> hdfs://host-10-17-81-192.coe.cloudera.com:8020/user/hive/warehouse/.hive-staging_hive_2015-10-29_19-04-08_375_5385987873542863570-3/-ext-10001
> to: /user/hive/data/yshi/nonexisting/test3
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename:
> hdfs://host-10-17-81-192.coe.cloudera.com:8020/user/hive/warehouse/.hive-staging_hive_2015-10-29_19-04-08_375_5385987873542863570-3/-ext-10001
> to: /user/hive/data/yshi/nonexisting/test3
> at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:101)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:209)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:144)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:212)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13231) Show helpful error message on failure to create table in nested directory
[
https://issues.apache.org/jira/browse/HIVE-13231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuben Kuhnert updated HIVE-13231:
--
Attachment: HIVE-13231.02.patch
> Show helpful error message on failure to create table in nested directory
> -
>
> Key: HIVE-13231
> URL: https://issues.apache.org/jira/browse/HIVE-13231
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
>Priority: Minor
> Attachments: HIVE-13231.01.patch, HIVE-13231.02.patch
>
>
> cannot store data in a directory whose parent doesn't exist, even though the
> target dir does have an existing ancestor on HDFS. This occurs when trying to
> perform {{create table }}.
> {code}
> 0: jdbc:hive2://10.17.81.192:1/default> create table test3 location
> '/user/hive/data/yshi/nonexisting/test3' as select * from sample_07;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> Error message:
> 2015-10-29 19:04:46,323 ERROR org.apache.hadoop.hive.ql.exec.Task: Failed
> with exception Unable to rename:
> hdfs://host-10-17-81-192.coe.cloudera.com:8020/user/hive/warehouse/.hive-staging_hive_2015-10-29_19-04-08_375_5385987873542863570-3/-ext-10001
> to: /user/hive/data/yshi/nonexisting/test3
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename:
> hdfs://host-10-17-81-192.coe.cloudera.com:8020/user/hive/warehouse/.hive-staging_hive_2015-10-29_19-04-08_375_5385987873542863570-3/-ext-10001
> to: /user/hive/data/yshi/nonexisting/test3
> at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:101)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:209)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1554)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1321)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1139)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:962)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:957)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:144)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:502)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:212)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13250) Compute predicate conversions on the client, instead of per row group
[ https://issues.apache.org/jira/browse/HIVE-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13250: --- Affects Version/s: 2.1.0 > Compute predicate conversions on the client, instead of per row group > - > > Key: HIVE-13250 > URL: https://issues.apache.org/jira/browse/HIVE-13250 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Ashutosh Chauhan > Attachments: HIVE-13250.patch > > > When running a query for the form > select count from table where ts_field = "2016-01-23 00:00:00"; > or > select count from table where ts_field = 1453507200 > ts_field is of type TIMESTAMP > The predicate is converted to whatever format is appropriate for TIMESTAMP > processing on each and every row group. > It would be far more efficient to process this once on the client - or even > once per task. > The same applies to ORC splt elimination as well - this is applied for each > stripe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189819#comment-15189819 ] Aihua Xu commented on HIVE-13251: - [~jarcec], [~ychena],[~ctang.ma] and [~szehon] can you take a look at the change? > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro definition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7672) Potential resource leak in EximUtil#createExportDump()
[
https://issues.apache.org/jira/browse/HIVE-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189774#comment-15189774
]
Hive QA commented on HIVE-7672:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670654/HIVE-7672.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7212/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7212/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7212/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
852be3b..f7c8fb5 branch-1 -> origin/branch-1
23d1558..1f9968a branch-2.0 -> origin/branch-2.0
0da77af..1e8a31e master -> origin/master
+ git reset --hard HEAD
HEAD is now at 0da77af HIVE-12039 : Temporarily disable TestSSL#testSSLVersion
(Ashutosh Chauhan via Thejas Nair)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 1e8a31e HIVE-13144 : HS2 can leak ZK ACL objects when curator
retries to create the persistent ephemeral node (Vaibhav Gumashta via Thejas
Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12670654 - PreCommit-HIVE-TRUNK-Build
> Potential resource leak in EximUtil#createExportDump()
> --
>
> Key: HIVE-7672
> URL: https://issues.apache.org/jira/browse/HIVE-7672
> Project: Hive
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: SUYEON LEE
>Priority: Minor
> Attachments: HIVE-7672.patch
>
>
> Here is related code:
> {code}
> OutputStream out = fs.create(metadataPath);
> out.write(jsonContainer.toString().getBytes("UTF-8"));
> out.close();
> {code}
> If out.write() throws exception, out would be left unclosed.
> out.close() should be enclosed in finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13236) LLAP: token renewal interval needs to be set
[
https://issues.apache.org/jira/browse/HIVE-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189763#comment-15189763
]
Hive QA commented on HIVE-13236:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792430/HIVE-13236.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9801 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7211/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7211/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12792430 - PreCommit-HIVE-TRUNK-Build
> LLAP: token renewal interval needs to be set
>
>
> Key: HIVE-13236
> URL: https://issues.apache.org/jira/browse/HIVE-13236
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13236.01.patch, HIVE-13236.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10632) Make sure TXN_COMPONENTS gets cleaned up if table is dropped before compaction.
[ https://issues.apache.org/jira/browse/HIVE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189740#comment-15189740 ] Alan Gates commented on HIVE-10632: --- +1 > Make sure TXN_COMPONENTS gets cleaned up if table is dropped before > compaction. > --- > > Key: HIVE-10632 > URL: https://issues.apache.org/jira/browse/HIVE-10632 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-10632.1.patch, HIVE-10632.2.patch, > HIVE-10632.3.patch, HIVE-10632.4.patch, HIVE-10632.5.patch, > HIVE-10632.6.patch, HIVE-10632.7.patch > > > The compaction process will clean up entries in TXNS, > COMPLETED_TXN_COMPONENTS, TXN_COMPONENTS. If the table/partition is dropped > before compaction is complete there will be data left in these tables. Need > to investigate if there are other situations where this may happen and > address it. > see HIVE-10595 for additional info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189735#comment-15189735 ] Alan Gates commented on HIVE-13201: --- The tests shouldn't be dropped, they should instead be changed so that they are properly transactional tables. The point of the tests is to cover that 'alter table compact' works from the command line. If you feel this is tested elsewhere in a better way then we can remove these tests. Other than that, +1 on the patch. > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12439) CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements
[ https://issues.apache.org/jira/browse/HIVE-12439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189710#comment-15189710 ] Wei Zheng commented on HIVE-12439: -- The only test failure doesn't seem related, and I cannot repro locally - TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping [~ekoifman] Can you review the patch? > CompactionTxnHandler.markCleaned() and TxnHandler.openTxns() misc improvements > -- > > Key: HIVE-12439 > URL: https://issues.apache.org/jira/browse/HIVE-12439 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-12439.1.patch > > > # add a safeguard to make sure IN clause is not too large; break up by txn id > to delete from TXN_COMPONENTS where tc_txnid in ... > # TxnHandler. openTxns() - use 1 insert with many rows in values() clause, > rather than 1 DB roundtrip per row -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13250) Compute predicate conversions on the client, instead of per row group
[ https://issues.apache.org/jira/browse/HIVE-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13250: Assignee: Ashutosh Chauhan Status: Patch Available (was: Open) > Compute predicate conversions on the client, instead of per row group > - > > Key: HIVE-13250 > URL: https://issues.apache.org/jira/browse/HIVE-13250 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Ashutosh Chauhan > Attachments: HIVE-13250.patch > > > When running a query for the form > select count from table where ts_field = "2016-01-23 00:00:00"; > or > select count from table where ts_field = 1453507200 > ts_field is of type TIMESTAMP > The predicate is converted to whatever format is appropriate for TIMESTAMP > processing on each and every row group. > It would be far more efficient to process this once on the client - or even > once per task. > The same applies to ORC splt elimination as well - this is applied for each > stripe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13250) Compute predicate conversions on the client, instead of per row group
[ https://issues.apache.org/jira/browse/HIVE-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13250: Attachment: HIVE-13250.patch > Compute predicate conversions on the client, instead of per row group > - > > Key: HIVE-13250 > URL: https://issues.apache.org/jira/browse/HIVE-13250 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth > Attachments: HIVE-13250.patch > > > When running a query for the form > select count from table where ts_field = "2016-01-23 00:00:00"; > or > select count from table where ts_field = 1453507200 > ts_field is of type TIMESTAMP > The predicate is converted to whatever format is appropriate for TIMESTAMP > processing on each and every row group. > It would be far more efficient to process this once on the client - or even > once per task. > The same applies to ORC splt elimination as well - this is applied for each > stripe. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12891) Hive fails when java.io.tmpdir is set to a relative location
[
https://issues.apache.org/jira/browse/HIVE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-12891:
---
Description:
The function {{SessionState.createSessionDirs}} fails when trying to create
directories where {{java.io.tmpdir}} is set to a relative location.
{code}
\[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED: IllegalArgumentException
java.net.URISyntaxException: Relative path in absolute URI:
file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
...
Minor variations:
\[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED: SemanticException
Exception while processing Exception while writing out the local file
o.a.h.hive.ql/parse.SemanticException: Exception while processing exception
while writing out local file
...
caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
at o.a.h.fs.Path.initialize (206)
at o.a.h.fs.Path.(197)...
at o.a.h.hive.ql.context.getScratchDir(267)
{code}
was:
The function {{SessionState.createSessionDirs}} fails when trying to create
directories where {{java.io.tmpdir}} is set to a relative location.
{code}
\[uber-SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED:
IllegalArgumentException java.net.URISyntaxException: Relative path in absolute
URI: file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
...
Minor variations:
\[uber-SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED: SemanticException
Exception while processing Exception while writing out the local file
o.a.h.hive.ql/parse.SemanticException: Exception while processing exception
while writing out local file
...
caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
at o.a.h.fs.Path.initialize (206)
at o.a.h.fs.Path.(197)...
at o.a.h.hive.ql.context.getScratchDir(267)
{code}
> Hive fails when java.io.tmpdir is set to a relative location
>
>
> Key: HIVE-12891
> URL: https://issues.apache.org/jira/browse/HIVE-12891
> Project: Hive
> Issue Type: Bug
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Attachments: HIVE-12891.01.19.2016.01.patch, HIVE-12891.03.patch,
> HIVE-12891.04.patch, HIVE-12981.01.22.2016.02.patch
>
>
> The function {{SessionState.createSessionDirs}} fails when trying to create
> directories where {{java.io.tmpdir}} is set to a relative location.
> {code}
> \[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED:
> IllegalArgumentException java.net.URISyntaxException: Relative path in
> absolute URI:
> file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
> ...
> Minor variations:
> \[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED: SemanticException
> Exception while processing Exception while writing out the local file
> o.a.h.hive.ql/parse.SemanticException: Exception while processing exception
> while writing out local file
> ...
> caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI:
> file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
> at o.a.h.fs.Path.initialize (206)
> at o.a.h.fs.Path.(197)...
> at o.a.h.hive.ql.context.getScratchDir(267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[
https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189684#comment-15189684
]
Sergio Peña commented on HIVE-13243:
Should we add some q-test on {{encryption_drop_partition.q}} for external
table/partitions as well?
> Hive drop table on encyption zone fails for external tables
> ---
>
> Key: HIVE-13243
> URL: https://issues.apache.org/jira/browse/HIVE-13243
> Project: Hive
> Issue Type: Bug
> Components: Encryption, Metastore
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13243.patch
>
>
> When dropping an external table with its data located in an encryption zone,
> hive should not throw out MetaException(message:Unable to drop table because
> it is in an encryption zone and trash is enabled. Use PURGE option to skip
> trash.) in checkTrashPurgeCombination since the data should not get deleted
> (or trashed) anyway regardless HDFS Trash is enabled or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13233) Use min and max values to estimate better stats for comparison operators
[ https://issues.apache.org/jira/browse/HIVE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189679#comment-15189679 ] Ashutosh Chauhan commented on HIVE-13233: - +1 > Use min and max values to estimate better stats for comparison operators > > > Key: HIVE-13233 > URL: https://issues.apache.org/jira/browse/HIVE-13233 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13233.01.patch, HIVE-13233.patch > > > We should benefit from the min/max values for each column to calculate more > precisely the number of rows produced by expressions with comparison operators -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13144) HS2 can leak ZK ACL objects when curator retries to create the persistent ephemeral node
[ https://issues.apache.org/jira/browse/HIVE-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-13144: - Resolution: Fixed Fix Version/s: 2.0.1 2.1.0 1.3.0 Status: Resolved (was: Patch Available) Patch committed to branch-1, master and branch-2.0 . Thanks for the patch [~vgumashta]! > HS2 can leak ZK ACL objects when curator retries to create the persistent > ephemeral node > > > Key: HIVE-13144 > URL: https://issues.apache.org/jira/browse/HIVE-13144 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.1, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-13144.1.patch, HIVE-13144.2.patch > > > When the node gets deleted from ZK due to connection loss and curator tries > to recreate the node, it might leak ZK ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13233) Use min and max values to estimate better stats for comparison operators
[ https://issues.apache.org/jira/browse/HIVE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13233: --- Attachment: HIVE-13233.01.patch > Use min and max values to estimate better stats for comparison operators > > > Key: HIVE-13233 > URL: https://issues.apache.org/jira/browse/HIVE-13233 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13233.01.patch, HIVE-13233.patch > > > We should benefit from the min/max values for each column to calculate more > precisely the number of rows produced by expressions with comparison operators -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13233) Use min and max values to estimate better stats for comparison operators
[ https://issues.apache.org/jira/browse/HIVE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13233 started by Jesus Camacho Rodriguez. -- > Use min and max values to estimate better stats for comparison operators > > > Key: HIVE-13233 > URL: https://issues.apache.org/jira/browse/HIVE-13233 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13233.01.patch, HIVE-13233.patch > > > We should benefit from the min/max values for each column to calculate more > precisely the number of rows produced by expressions with comparison operators -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13233) Use min and max values to estimate better stats for comparison operators
[ https://issues.apache.org/jira/browse/HIVE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13233: --- Status: Open (was: Patch Available) > Use min and max values to estimate better stats for comparison operators > > > Key: HIVE-13233 > URL: https://issues.apache.org/jira/browse/HIVE-13233 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13233.01.patch, HIVE-13233.patch > > > We should benefit from the min/max values for each column to calculate more > precisely the number of rows produced by expressions with comparison operators -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[ https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189611#comment-15189611 ] Chaoyu Tang commented on HIVE-13243: Patch has been uploaded to https://reviews.apache.org/r/44664/ for review. [~spena] could you take a look, thanks. > Hive drop table on encyption zone fails for external tables > --- > > Key: HIVE-13243 > URL: https://issues.apache.org/jira/browse/HIVE-13243 > Project: Hive > Issue Type: Bug > Components: Encryption, Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13243.patch > > > When dropping an external table with its data located in an encryption zone, > hive should not throw out MetaException(message:Unable to drop table because > it is in an encryption zone and trash is enabled. Use PURGE option to skip > trash.) in checkTrashPurgeCombination since the data should not get deleted > (or trashed) anyway regardless HDFS Trash is enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[ https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-13243: --- Attachment: HIVE-13243.patch > Hive drop table on encyption zone fails for external tables > --- > > Key: HIVE-13243 > URL: https://issues.apache.org/jira/browse/HIVE-13243 > Project: Hive > Issue Type: Bug > Components: Encryption, Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13243.patch > > > When dropping an external table with its data located in an encryption zone, > hive should not throw out MetaException(message:Unable to drop table because > it is in an encryption zone and trash is enabled. Use PURGE option to skip > trash.) in checkTrashPurgeCombination since the data should not get deleted > (or trashed) anyway regardless HDFS Trash is enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13243) Hive drop table on encyption zone fails for external tables
[ https://issues.apache.org/jira/browse/HIVE-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-13243: --- Status: Patch Available (was: Open) > Hive drop table on encyption zone fails for external tables > --- > > Key: HIVE-13243 > URL: https://issues.apache.org/jira/browse/HIVE-13243 > Project: Hive > Issue Type: Bug > Components: Encryption, Metastore >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-13243.patch > > > When dropping an external table with its data located in an encryption zone, > hive should not throw out MetaException(message:Unable to drop table because > it is in an encryption zone and trash is enabled. Use PURGE option to skip > trash.) in checkTrashPurgeCombination since the data should not get deleted > (or trashed) anyway regardless HDFS Trash is enabled or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13251: Status: Patch Available (was: Open) > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro definition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189444#comment-15189444 ] Aihua Xu commented on HIVE-13251: - Attached the patch-1: when we read the file achema, we will try to read and convert the string to int for decimal scale, but we disallow setting precision and scale to string when hive creates the avro file. Added the positive and negative unit tests. > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro definition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13251: Description: HIVE-7174 makes the avro schema change to match avro definition, while it breaks the compatibility if the file is generated from the previous Hive although the file schema from the file for such decimal is not correct based on avro defninition. We should allow to read old file format "precision" : "4", "scale": "8", but when we write, we should write in the new format. (was: HIVE-7174 makes the avro schema change to match avro definition, while it breaks the compatibility if the file is generated from the previous Hive. We should allow to read old file format "precision" : "4", "scale": "8", but write in the new format.) > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro defninition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13251) hive can't read the decimal in AVRO file generated from previous version
[ https://issues.apache.org/jira/browse/HIVE-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13251: Attachment: HIVE-13251.1.patch > hive can't read the decimal in AVRO file generated from previous version > > > Key: HIVE-13251 > URL: https://issues.apache.org/jira/browse/HIVE-13251 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13251.1.patch > > > HIVE-7174 makes the avro schema change to match avro definition, while it > breaks the compatibility if the file is generated from the previous Hive > although the file schema from the file for such decimal is not correct based > on avro definition. We should allow to read old file format "precision" : > "4", "scale": "8", but when we write, we should write in the new format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladyslav Pavlenko updated HIVE-10176:
--
Assignee: Vladyslav Pavlenko
Affects Version/s: (was: 1.0.0)
2.0.0
Status: Patch Available (was: Open)
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Attachments: HIVE-10176.1.patch, HIVE-10176.2.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13144) HS2 can leak ZK ACL objects when curator retries to create the persistent ephemeral node
[
https://issues.apache.org/jira/browse/HIVE-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189245#comment-15189245
]
Hive QA commented on HIVE-13144:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792085/HIVE-13144.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9786 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7210/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7210/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12792085 - PreCommit-HIVE-TRUNK-Build
> HS2 can leak ZK ACL objects when curator retries to create the persistent
> ephemeral node
>
>
> Key: HIVE-13144
> URL: https://issues.apache.org/jira/browse/HIVE-13144
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-13144.1.patch, HIVE-13144.2.patch
>
>
> When the node gets deleted from ZK due to connection loss and curator tries
> to recreate the node, it might leak ZK ACL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values
[
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladyslav Pavlenko updated HIVE-10176:
--
Attachment: HIVE-10176.2.patch
I changed regex in view of bucketed tables.
> skip.header.line.count causes values to be skipped when performing insert
> values
>
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Wenbo Wang
> Attachments: HIVE-10176.1.patch, HIVE-10176.2.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES
> ("skip.header.line.count"="1") the first value listed is also skipped.
> create table test (row int, name string) TBLPROPERTIES
> ("skip.header.line.count"="1");
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13217) Replication for HoS mapjoin small file needs to respect dfs.replication.max
[ https://issues.apache.org/jira/browse/HIVE-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-13217: --- Assignee: Chinna Rao Lalam > Replication for HoS mapjoin small file needs to respect dfs.replication.max > --- > > Key: HIVE-13217 > URL: https://issues.apache.org/jira/browse/HIVE-13217 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Assignee: Chinna Rao Lalam >Priority: Minor > > Currently Hive on Spark Mapjoin replicates small table file to a hard-coded > value of 10. See SparkHashTableSinkOperator.MIN_REPLICATION. > When dfs.replication.max is less than 10, HoS query fails. This constant > should cap at dfs.replication.max. > Normally dfs.replication.max seems set at 512. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Barr updated HIVE-11484: --- Status: Patch Available (was: In Progress) > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.01.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-11484 started by Deepak Barr. -- > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.01.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-11484 stopped by Deepak Barr. -- > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.01.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-11484 started by Deepak Barr. -- > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.01.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Barr updated HIVE-11484: --- Attachment: HIVE-11484.01.patch > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.01.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Barr updated HIVE-11484: --- Attachment: (was: HIVE-11484.master.patch) > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13233) Use min and max values to estimate better stats for comparison operators
[
https://issues.apache.org/jira/browse/HIVE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188984#comment-15188984
]
Hive QA commented on HIVE-13233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12792069/HIVE-13233.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9801 tests executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_precision
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7209/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7209/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7209/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12792069 - PreCommit-HIVE-TRUNK-Build
> Use min and max values to estimate better stats for comparison operators
>
>
> Key: HIVE-13233
> URL: https://issues.apache.org/jira/browse/HIVE-13233
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13233.patch
>
>
> We should benefit from the min/max values for each column to calculate more
> precisely the number of rows produced by expressions with comparison operators
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11484) Fix ObjectInspector for Char and VarChar
[ https://issues.apache.org/jira/browse/HIVE-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Barr updated HIVE-11484: --- Status: Open (was: Patch Available) > Fix ObjectInspector for Char and VarChar > > > Key: HIVE-11484 > URL: https://issues.apache.org/jira/browse/HIVE-11484 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Amareshwari Sriramadasu >Assignee: Deepak Barr > Attachments: HIVE-11484.master.patch > > > The creation of HiveChar and Varchar is not happening through ObjectInspector. > Here is fix we pushed internally : > https://github.com/InMobi/hive/commit/fe95c7850e7130448209141155f28b25d3504216 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
