[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255976#comment-15255976
]
Nemon Lou commented on HIVE-13602:
--
I just copy it from the result of "explain rewrite q16".
{noformat}
explain rewrite
select
p_brand,
p_type,
p_size,
count(distinct ps_suppkey) as supplier_cnt
from
partsupp,
part
where
p_partkey = ps_partkey
and p_brand <> 'Brand#34'
and p_type not like 'ECONOMY BRUSHED%'
and p_size in (22, 14, 27, 49, 21, 33, 35, 28)
and partsupp.ps_suppkey not in (
select
s_suppkey
from
supplier
where
s_comment like '%Customer%Complaints%'
)
group by
p_brand,
p_type,
p_size
order by
supplier_cnt desc,
p_brand,
p_type,
p_size;
{noformat}
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: calcite_cbo_bad.out, calcite_cbo_good.out,
> explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out,
> explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255974#comment-15255974 ] Prasanth Jayachandran commented on HIVE-13536: -- [~sseth] Could you please take another look? > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255973#comment-15255973 ] Prasanth Jayachandran commented on HIVE-13097: -- LGMT, +1 > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255964#comment-15255964
]
Nemon Lou commented on HIVE-13602:
--
>From calcite rel plan ,we can see the p_brand column is declared to fetch.
And the value for p_brand column is zero in the final result.
{noformat}
0: jdbc:hive2://189.39.151.141:21066/> select * from q16_cbo_bad limit 1;
++---+---+-+--+
| q16_cbo_bad.p_brand | q16_cbo_bad.p_type | q16_cbo_bad.p_size |
q16_cbo_bad.supplier_cnt |
++---+---+-+--+
| 0 | MEDIUM BRUSHED TIN| 21| 298
|
++---+---+-+--+
1 row selected (0.139 seconds)
0: jdbc:hive2://189.39.151.141:21066/> select * from q16_cbo_rewrite limit 1;
++---+---+-+--+
| q16_cbo_rewrite.p_brand | q16_cbo_rewrite.p_type | q16_cbo_rewrite.p_size
| q16_cbo_rewrite.supplier_cnt |
++---+---+-+--+
| Brand#31 | PROMO ANODIZED NICKEL | 14
| 40 |
++---+---+-+--+
1 row selected (0.961 seconds)
{noformat}
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: calcite_cbo_bad.out, calcite_cbo_good.out,
> explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out,
> explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255953#comment-15255953
]
Pengcheng Xiong commented on HIVE-13602:
what is {code} 1 = 1 {code} in the where condition?
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: calcite_cbo_bad.out, calcite_cbo_good.out,
> explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out,
> explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255950#comment-15255950 ] Pengcheng Xiong commented on HIVE-13602: [~nemon], if p_brand is missing, i would expect to see that one column is missing, rather than different number of rows? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255937#comment-15255937 ] Nemon Lou commented on HIVE-13602: -- The join order seems to be the same for both queries.The difference is that the bad plan only select three columns for table part at stage-1 (p_brand is missing),the good one select four. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[
https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255934#comment-15255934
]
Hive QA commented on HIVE-13563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800332/HIVE-13563.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9992 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/70/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-70/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800332 - PreCommit-HIVE-MASTER-Build
> Hive Streaming does not honor orc.compress.size and orc.stripe.size table
> properties
>
>
> Key: HIVE-13563
> URL: https://issues.apache.org/jira/browse/HIVE-13563
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: TODOC2.1
> Attachments: HIVE-13563.1.patch
>
>
> According to the doc:
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax
> One should be able to specify tblproperties for many ORC options.
> But the settings for orc.compress.size and orc.stripe.size don't take effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13602: - Attachment: explain_cbo_good(rewrite)_part3.out explain_cbo_good(rewrite)_part2.out explain_cbo_good(rewrite)_part1.out calcite_cbo_good.out calcite_cbo_bad.out > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: calcite_cbo_bad.out, calcite_cbo_good.out, > explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out, explain_cbo_good(rewrite)_part1.out, > explain_cbo_good(rewrite)_part2.out, explain_cbo_good(rewrite)_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13603) Fix ptest unit tests broken by HIVE13505
[ https://issues.apache.org/jira/browse/HIVE-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13603: -- Status: Patch Available (was: Open) > Fix ptest unit tests broken by HIVE13505 > > > Key: HIVE-13603 > URL: https://issues.apache.org/jira/browse/HIVE-13603 > Project: Hive > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13603.01.patch > > > HIVE-13505 broke some unit tests in the ptest2 framework, which need to be > fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13603) Fix ptest unit tests broken by HIVE13505
[ https://issues.apache.org/jira/browse/HIVE-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13603: -- Attachment: HIVE-13603.01.patch Simple fix. cc [~szehon], [~spena], [~ashutoshc] for review. Don't think this needs to take up a slot in the build queue. Submitting anyway for now. Will commit and cancel the precommit run if people are ok with that. There's still some failing tests after this, but those exist without HIVE-13505 as well. I suspect that has something to do with enabling / disabling the spark profile. > Fix ptest unit tests broken by HIVE13505 > > > Key: HIVE-13603 > URL: https://issues.apache.org/jira/browse/HIVE-13603 > Project: Hive > Issue Type: Task >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13603.01.patch > > > HIVE-13505 broke some unit tests in the ptest2 framework, which need to be > fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255921#comment-15255921
]
Pengcheng Xiong commented on HIVE-13602:
[~nemon], could u also post the explain plan for the rewrite query? As I can
see from the attachement, the original plan when CBO is on is to
{code}
(join part with supplier cnt=0) join (left outer join partsupp with supplier).
{code}
I would like to see if the rewrite query follows the same plan.
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255914#comment-15255914
]
Nemon Lou commented on HIVE-13602:
--
Another information : after rewrite q16 into the following form,CBO can return
right result:
{noformat}
select
p_brand,
p_type,
p_size,
count(distinct ps_suppkey) as supplier_cnt
from
partsupp
left outer join (
select
s_suppkey
from
supplier
where s_comment like '%Customer%Complaints%'
) sq_1 on partsupp.ps_suppkey = sq_1.s_suppkey
join
( select count(*) as cn from (select * from supplier where s_comment like
'%Customer%Complaints%' ) b where s_suppkey is null) a
on a.cn= 0
join part
where
p_partkey = ps_partkey
and p_brand <> 'Brand#34'
and p_type not like 'ECONOMY BRUSHED%'
and p_size in (22, 14, 27, 49, 21, 33, 35, 28)
and 1 = 1 and sq_1.s_suppkey is null
group by
p_brand,
p_type,
p_size
order by
supplier_cnt desc,
p_brand,
p_type,
p_size;
{noformat}
> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
>Affects Versions: 1.2.1
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: explain_cbo_bad_part1.out, explain_cbo_bad_part2.out,
> explain_cbo_bad_part3.out
>
>
> Running tpch with factor 2,
> q16 returns 1,160 rows when CBO is on,
> while returns 59,616 rows when CBO is off.
> See attachment for detail .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13568: Attachment: HIVE-13568.1.patch > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch, HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13568: Status: Patch Available (was: Open) > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch, HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12959) LLAP: Add task scheduler timeout when no nodes are alive
[ https://issues.apache.org/jira/browse/HIVE-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12959: -- Attachment: HIVE-12959.7.patch Re-uploading for a jenkins run. > LLAP: Add task scheduler timeout when no nodes are alive > > > Key: HIVE-12959 > URL: https://issues.apache.org/jira/browse/HIVE-12959 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12959.1.patch, HIVE-12959.2.patch, > HIVE-12959.3.patch, HIVE-12959.5.patch, HIVE-12959.6.patch, > HIVE-12959.6.patch, HIVE-12959.7.patch, HIVE-12959.7.patch, HIVE-12959.7.txt > > > When there are no llap daemons running task scheduler should have a timeout > to fail the query instead of waiting forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13568) Add UDFs to support column-masking
[ https://issues.apache.org/jira/browse/HIVE-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13568: Status: Open (was: Patch Available) > Add UDFs to support column-masking > -- > > Key: HIVE-13568 > URL: https://issues.apache.org/jira/browse/HIVE-13568 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj > Attachments: HIVE-13568.1.patch > > > HIVE-13125 added support to provide column-masking and row-filtering during > select via HiveAuthorizer interface. This JIRA is track addition of UDFs that > can be used by HiveAuthorizer implementations to mask column values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255910#comment-15255910 ] Siddharth Seth commented on HIVE-13097: --- Test failures are unrelated. > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch, > HIVE-13097.03.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13537) Update slf4j version to 1.7.10
[ https://issues.apache.org/jira/browse/HIVE-13537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13537: -- Attachment: HIVE-13537.01.patch > Update slf4j version to 1.7.10 > -- > > Key: HIVE-13537 > URL: https://issues.apache.org/jira/browse/HIVE-13537 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Minor > Attachments: HIVE-13537.01.patch, HIVE-13537.01.patch > > > Both Hadoop and Tez are on 1.7.10, and have been for a while. We should > update hive to use this version as well. Should get rid of some of the noise > in the logs related to multiple version.s -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HIVE-13602: - Attachment: explain_cbo_bad_part3.out explain_cbo_bad_part2.out explain_cbo_bad_part1.out I have run "analyze table .. for columns" for all the tables. Splitting the explain file into 3 parts in order to upload. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > Attachments: explain_cbo_bad_part1.out, explain_cbo_bad_part2.out, > explain_cbo_bad_part3.out > > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13469) LLAP: Support delayed scheduling for locality
[ https://issues.apache.org/jira/browse/HIVE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13469: -- Attachment: HIVE-13469.02.patch Re-uploading for precommit. > LLAP: Support delayed scheduling for locality > - > > Key: HIVE-13469 > URL: https://issues.apache.org/jira/browse/HIVE-13469 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13469.01.patch, HIVE-13469.02.patch, > HIVE-13469.02.patch > > > LLAP currently supports forcing locality. Change this to support a time based > delay for locality as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255900#comment-15255900 ] Pengcheng Xiong commented on HIVE-13602: [~nemon], do you have all the table/column stats for all the tables? Could u attach the explain results for CBO on and off? Thanks. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255897#comment-15255897 ] Nemon Lou commented on HIVE-13602: -- I used MR engine. The attachment is bigger than 50K ,and has been rejected by firewall or something else. > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-13602: -- Assignee: Pengcheng Xiong > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou >Assignee: Pengcheng Xiong > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on
[ https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255892#comment-15255892 ] Pengcheng Xiong commented on HIVE-13602: cc'ing [~ashutoshc] and [~hagleitn]. [~nemon], could u be more specific about the execution engine that you used? MR or Tez? > TPCH q16 return wrong result when CBO is on > --- > > Key: HIVE-13602 > URL: https://issues.apache.org/jira/browse/HIVE-13602 > Project: Hive > Issue Type: Bug > Components: CBO, Logical Optimizer >Affects Versions: 1.2.1 >Reporter: Nemon Lou > > Running tpch with factor 2, > q16 returns 1,160 rows when CBO is on, > while returns 59,616 rows when CBO is off. > See attachment for detail . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[
https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255873#comment-15255873
]
Hive QA commented on HIVE-12837:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800311/HIVE-12837.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 9869 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestHiveMetaStoreWithEnvironmentContext.testEnvironmentContext
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException
org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure
org.apache.hive.hcatalog.listener.TestDbNotificationListener.drop
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Status: Patch Available (was: Open) Resubmitting for jenkins to pick > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13421) Propagate job progress in operation status
[ https://issues.apache.org/jira/browse/HIVE-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated HIVE-13421: --- Fix Version/s: 2.1.0 Status: Open (was: Patch Available) > Propagate job progress in operation status > -- > > Key: HIVE-13421 > URL: https://issues.apache.org/jira/browse/HIVE-13421 > Project: Hive > Issue Type: Improvement >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13421.01.patch, HIVE-13421.02.patch, > HIVE-13421.03.patch, HIVE-13421.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13525) HoS hangs when job is empty
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13525: -- Attachment: HIVE-13525.2.patch Upload same patch to run tests. > HoS hangs when job is empty > --- > > Key: HIVE-13525 > URL: https://issues.apache.org/jira/browse/HIVE-13525 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-13525.1.patch, HIVE-13525.2.patch, > HIVE-13525.2.patch > > > Observed in local tests. This should be the cause of HIVE-13402. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10293) enabling travis-ci build?
[ https://issues.apache.org/jira/browse/HIVE-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255791#comment-15255791 ] Gabor Liptak commented on HIVE-10293: - I'm still stumped with this ... I added a new repository with the correct Maven URL, but it is not getting used. https://repository.apache.org/service/local/repositories/releases/content/ https://travis-ci.org/gliptak/hive/builds/125466372 [INFO] [INFO] [INFO] Building Hive Service 2.1.0-SNAPSHOT [INFO] Downloading: https://oss.sonatype.org/content/repositories/releases/org/apache/directory/client/ldap/ldap-client-api/1.0.0-M14/ldap-client-api-1.0.0-M14.pom Downloading: https://repository.apache.org/releases/org/apache/directory/client/ldap/ldap-client-api/1.0.0-M14/ldap-client-api-1.0.0-M14.pom [WARNING] The POM for org.apache.directory.client.ldap:ldap-client-api:jar:1.0.0-M14 is missing, no dependency information available Downloading: https://nexus.codehaus.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/maven-metadata.xml [WARNING] Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to codehaus-snapshots (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org [WARNING] Failure to transfer org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from https://nexus.codehaus.org/snapshots/ was cached in the local repository, resolution will not be reattempted until the update interval of codehaus-snapshots has elapsed or updates are forced. Original error: Could not transfer metadata org.apache.directory.client.ldap:ldap-client-api:0.1-SNAPSHOT/maven-metadata.xml from/to codehaus-snapshots (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org Downloading: https://repository.apache.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom Downloading: https://nexus.codehaus.org/snapshots/org/apache/directory/client/ldap/ldap-client-api/0.1-SNAPSHOT/ldap-client-api-0.1-SNAPSHOT.pom > enabling travis-ci build? > - > > Key: HIVE-10293 > URL: https://issues.apache.org/jira/browse/HIVE-10293 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HIVE-10293.1.patch, HIVE-10293.2.diff > > > I would like to contribute a .travis.yml for Hive. > In particular, this would allow contributors working through Github, to > validate their own commits on their own branches. > Please comment. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wataru Yukawa resolved HIVE-13239.
--
Resolution: Fixed
> "java.lang.OutOfMemoryError: unable to create new native thread" occurs at
> Hive on Tez
> --
>
> Key: HIVE-13239
> URL: https://issues.apache.org/jira/browse/HIVE-13239
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Environment: HDP2.3.4
> JDK1.8
> CentOS 6
>Reporter: Wataru Yukawa
>
> "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000
> HiveServer2 memory leak occurs.
> hive query
> {code}
> FROM hoge_tmp
> INSERT INTO TABLE hoge PARTITION (...)
>SELECT ... WHERE ...
> {code}
> stacktrace
> {code}
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577)
> at
> org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310)
> at
> org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559)
> at org.apache.tez.client.TezClient.start(TezClient.java:395)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java
[jira] [Commented] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255788#comment-15255788
]
Wataru Yukawa commented on HIVE-13239:
--
After upgrading to HDP2.4, this problem doesn't occur.
So, I close this issue.
Thanks.
> "java.lang.OutOfMemoryError: unable to create new native thread" occurs at
> Hive on Tez
> --
>
> Key: HIVE-13239
> URL: https://issues.apache.org/jira/browse/HIVE-13239
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Environment: HDP2.3.4
> JDK1.8
> CentOS 6
>Reporter: Wataru Yukawa
>
> "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000
> HiveServer2 memory leak occurs.
> hive query
> {code}
> FROM hoge_tmp
> INSERT INTO TABLE hoge PARTITION (...)
>SELECT ... WHERE ...
> {code}
> stacktrace
> {code}
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577)
> at
> org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310)
> at
> org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559)
> at org.apache.tez.client.TezClient.start(TezClient.java:395)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hiv
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255761#comment-15255761
]
Hive QA commented on HIVE-7693:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12771809/HIVE-7693.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/67/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/67/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-67/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-67/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
a42bc67..413645d master -> origin/master
+ git reset --hard HEAD
HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not
exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 413645d HIVE-13570: Some queries with Union all fail when CBO is
off (Yongzhi Chen, reviewed by Chaoyu Tang)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12771809 - PreCommit-HIVE-MASTER-Build
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch, HIVE-7693.05.patch,
> HIVE-7693.06.patch, HIVE-7693.07.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13346) LLAP doesn't update metadata priority when reusing from cache; some tweaks in LRFU policy
[
https://issues.apache.org/jira/browse/HIVE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255759#comment-15255759
]
Hive QA commented on HIVE-13346:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800298/HIVE-13346.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 57 failed/errored test(s), 9943 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.
[jira] [Commented] (HIVE-10293) enabling travis-ci build?
[ https://issues.apache.org/jira/browse/HIVE-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255756#comment-15255756 ] Gabor Liptak commented on HIVE-10293: - https://docs.travis-ci.com/user/ci-environment/ > enabling travis-ci build? > - > > Key: HIVE-10293 > URL: https://issues.apache.org/jira/browse/HIVE-10293 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HIVE-10293.1.patch, HIVE-10293.2.diff > > > I would like to contribute a .travis.yml for Hive. > In particular, this would allow contributors working through Github, to > validate their own commits on their own branches. > Please comment. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10293) enabling travis-ci build?
[ https://issues.apache.org/jira/browse/HIVE-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255754#comment-15255754 ] Gabor Liptak commented on HIVE-10293: - https://github.com/travis-ci/travis-ci/issues/2677 > enabling travis-ci build? > - > > Key: HIVE-10293 > URL: https://issues.apache.org/jira/browse/HIVE-10293 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Gabor Liptak >Assignee: Gabor Liptak >Priority: Minor > Attachments: HIVE-10293.1.patch, HIVE-10293.2.diff > > > I would like to contribute a .travis.yml for Hive. > In particular, this would allow contributors working through Github, to > validate their own commits on their own branches. > Please comment. > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some queries with Union all fail when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13570:
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Confirmed the Llap failures are not related for precommit build #55 for
HIVE-13494 has the exact same failures.
Committed to master branch.
> Some queries with Union all fail when CBO is off
>
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Fix For: 2.1.0
>
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[
https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255731#comment-15255731
]
Hive QA commented on HIVE-11887:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800281/HIVE-11887.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 9952 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/65/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/65/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-65/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800281 - PreCommit-HIVE-MASTER-Build
> spark tests break
[jira] [Commented] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[
https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255710#comment-15255710
]
Hive QA commented on HIVE-13391:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800278/HIVE-13391.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/64/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/64/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-64/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-64/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not
exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not
exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800278 - PreCommit-HIVE-MASTER-Build
> add an option to LLAP to use keytab to authenticate to read data
>
>
> Key: HIVE-13391
> URL: https://issues.apache.org/jira/browse/HIVE-13391
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch,
> HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch,
> HIVE-13391.patch
>
>
> This can be used for non-doAs case to allow access to clients who don't
> propagate HDFS tokens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[
https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255709#comment-15255709
]
Hive QA commented on HIVE-13449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800357/HIVE-13449.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 9900 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_varchar_4.q-smb_cache.q-tez_join_hash.q-and-8-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-vector_decimal_aggregate.q-tez_self_join.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testSimpleTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/63/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/63/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-63/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests ex
[jira] [Updated] (HIVE-13601) Deadline is not registered at some places in HiveMetastore before the ObjectStore/MetaStoreDirectSql APIs are invoked
[
https://issues.apache.org/jira/browse/HIVE-13601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13601:
-
Summary: Deadline is not registered at some places in HiveMetastore before
the ObjectStore/MetaStoreDirectSql APIs are invoked (was: Deadline is not
registered at some places in HiveMetastore before the
ObjectStore/MetaStoreDirectSQL APIs are invoked)
> Deadline is not registered at some places in HiveMetastore before the
> ObjectStore/MetaStoreDirectSql APIs are invoked
> -
>
> Key: HIVE-13601
> URL: https://issues.apache.org/jira/browse/HIVE-13601
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>
> HIVE-13542 exposed this issue as shown below when making Metastore API calls
> via CBO code path.
> {code}
> 16-04-21T11:45:55,201 DEBUG [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]:
> metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:timingTrace(864)) -
> Direct SQL query in 14.825303ms + 0.086336ms, the query is [select
> "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE",
> "DOUBLE_LOW_VALU\
> E", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", "BIG_DECIMAL_HIGH_VALUE",
> "NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", "MAX_COL_LEN", "NUM_TRUES",
> "NUM_FALSES", "LAST_ANALYZED" from "TAB_COL_STATS" where "DB_NAME" = ? and
> "TABLE_NAME" = ? and "COLUMN_NAME" in (...)]
> 2016-04-21T11:45:55,242 WARN [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]:
> metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(2602)) - Direct
> SQL failed, falling back to ORM
> org.apache.hadoop.hive.metastore.api.MetaException: The threadlocal Deadline
> is null, please register it first.
> at
> org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:178)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:149)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:1717)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:1201)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6849)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6846)
> ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2567)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6845)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6839)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4361)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1613)
> [hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:347)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:3317)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:758)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:302)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:417)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:410)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:158)
> [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hi
[jira] [Commented] (HIVE-13239) "java.lang.OutOfMemoryError: unable to create new native thread" occurs at Hive on Tez
[
https://issues.apache.org/jira/browse/HIVE-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255665#comment-15255665
]
Takuma Wakamori commented on HIVE-13239:
Does this issue already resolved?
([~wyukawa] mentioned at HIVE-13273 that the OOM Error does not occur after
upgrading to HDP2.4.)
If not, I'll address this issue.
I'm trying to fix bugs around hiveserver2.
Thanks.
> "java.lang.OutOfMemoryError: unable to create new native thread" occurs at
> Hive on Tez
> --
>
> Key: HIVE-13239
> URL: https://issues.apache.org/jira/browse/HIVE-13239
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Environment: HDP2.3.4
> JDK1.8
> CentOS 6
>Reporter: Wataru Yukawa
>
> "ps -L $(pgrep -f hiveserver2) | wc -l" is more than 15,000
> HiveServer2 memory leak occurs.
> hive query
> {code}
> FROM hoge_tmp
> INSERT INTO TABLE hoge PARTITION (...)
>SELECT ... WHERE ...
> {code}
> stacktrace
> {code}
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391)
> at
> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.start(DFSOutputStream.java:2238)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1753)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:577)
> at
> org.apache.tez.common.TezCommonUtils.createFileForAM(TezCommonUtils.java:310)
> at
> org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:559)
> at org.apache.tez.client.TezClient.start(TezClient.java:395)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execu
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Attachment: HIVE-13341.07.patch > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[
https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255649#comment-15255649
]
Hive QA commented on HIVE-9660:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800288/HIVE-9660.10.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 73 failed/errored test(s), 9947 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_file_dump
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_lengths
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge12
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge12
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.TestTxnCommands2.testBucketizedInputFormat
org.apache.hadoop.hive.ql.TestTxnCommands2.testDeleteIn
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcPPD
org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testBloomFilter2
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDictionaryThreshold
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDump
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.q
[jira] [Commented] (HIVE-13592) metastore calls map is not thread safe
[
https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255608#comment-15255608
]
Hive QA commented on HIVE-13592:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800271/HIVE-13592.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/61/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/61/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-61/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-61/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not
exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
Removing pom.xml.orig
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not
exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800271 - PreCommit-HIVE-MASTER-Build
> metastore calls map is not thread safe
> --
>
> Key: HIVE-13592
> URL: https://issues.apache.org/jira/browse/HIVE-13592
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13592.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[
https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255607#comment-15255607
]
Hive QA commented on HIVE-13097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800265/HIVE-13097.03.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 9944 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropTable
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/60/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/60/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-60/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800265 - PreCommit-HIVE-MASTER-Build
> [Umbrella] Changes dependent on Tez 0.8.3
> -
>
> Key: HIVE-13097
> URL: https://issues.apache.org/jira/browse/HIVE-13097
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
> Attachments: HIVE-13097.01.patch, HIVE-13097.02.patch,
> HIVE-13097.03.patch, HIVE-13097.versonChange.patch
>
>
[jira] [Commented] (HIVE-13586) Allow UDFs/UDTFs to indicate whether they can run with fetch task conversion
[
https://issues.apache.org/jira/browse/HIVE-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255573#comment-15255573
]
Hive QA commented on HIVE-13586:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800261/HIVE-13586.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 9922 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-schema_evol_text_nonvec_mapwork_table.q-vector_groupby_reduce.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-join1.q-mapjoin_decimal.q-vectorized_distinct_gby.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-auto_sortmerge_join_7.q-orc_merge9.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alias_casted_column
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_coalesce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_named_struct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_struct
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertPartition
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/59/testReport
Console output:
http://ec2-54-177
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: (was: HIVE-12878.09.patch) > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12878) Support Vectorization for TEXTFILE and other formats
[ https://issues.apache.org/jira/browse/HIVE-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-12878: Attachment: HIVE-12878.09.patch > Support Vectorization for TEXTFILE and other formats > > > Key: HIVE-12878 > URL: https://issues.apache.org/jira/browse/HIVE-12878 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-12878.01.patch, HIVE-12878.02.patch, > HIVE-12878.03.patch, HIVE-12878.04.patch, HIVE-12878.05.patch, > HIVE-12878.06.patch, HIVE-12878.07.patch, HIVE-12878.08.patch, > HIVE-12878.09.patch > > > Support vectorizing when the input format is TEXTFILE and other formats for > better Map Vertex performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13258) LLAP: Add hdfs bytes read and spilled bytes to tez print summary
[
https://issues.apache.org/jira/browse/HIVE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255541#comment-15255541
]
Hive QA commented on HIVE-13258:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800260/HIVE-13258.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/58/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/58/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-58/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-58/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
7419768..a42bc67 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 7419768 HIVE-13527: Using deprecated APIs in HBase client causes
zookeeper connection leaks (Naveen Gangam via Chaoyu Tang)
+ git clean -f -d
Removing
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleConstantReduction.java
Removing
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregatePullUpConstantsRule.java
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at a42bc67 HIVE-13494: LLAP: Some metrics from daemon are not
exposed to hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800260 - PreCommit-HIVE-MASTER-Build
> LLAP: Add hdfs bytes read and spilled bytes to tez print summary
>
>
> Key: HIVE-13258
> URL: https://issues.apache.org/jira/browse/HIVE-13258
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-13258.1.patch, HIVE-13258.1.patch,
> llap-fs-counters-full-cache-hit.png, llap-fs-counters.png
>
>
> When printing counters to console it will be useful to print hdfs bytes read
> and spilled bytes which will help with debugging issues faster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255540#comment-15255540
]
Hive QA commented on HIVE-13068:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800258/HIVE-13068.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 263 failed/errored test(s), 9930 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-mapjoin_mapjoin.q-update_all_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguitycheck
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cast_on_constant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_SortUnionTransposeRule
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_pad_convert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropWhen
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_sel
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_colname
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_cond_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_eq_with_case_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_folder_predicate
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_foldts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_duplicate_key
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_unused
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_interval_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDri
[jira] [Updated] (HIVE-13176) OutOfMemoryError : GC overhead limit exceeded
[ https://issues.apache.org/jira/browse/HIVE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuma Wakamori updated HIVE-13176: --- Description: KDetected leaks while testing hiveserver2 concurrency setup with LLAP. 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]: operation.Operation (SQLOperation.java:run(230)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. GC overhead limit exceeded at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:333) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:177) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:227) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_45] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_45] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.1.2.3.5.1-36.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:239) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45] was: Detected leaks while testing hiveserver2 concurrency setup with LLAP. 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]: operation.Operation (SQLOperation.java:run(230)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.StatsTask. GC overhead limit exceeded at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:333) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:177) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73) ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:227) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_45] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_45] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.1.2.3.5.1-36.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:239) [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45] > OutOfMemoryError : GC overhead limit exceeded > -- > > Key: HIVE-13176 > URL: https://issues.apache.org/jira/browse/HIVE-13176 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Kavan Suresh >Assignee: Siddharth Seth > Attachments: dataNucleus.png, fs.png, shutdownhook.png > > > KDetected leaks while testing hiveserver2 concurrency setup with LLAP. > 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]: > operation.Operation (SQLOperation.java:run(230)) - Error running hive query: > org.apache.hive.service.cli.HiveSQLException: Error while processing > statement: FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.StatsTask. GC over
[jira] [Commented] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[
https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255533#comment-15255533
]
Lefty Leverenz commented on HIVE-13494:
---
Doc note: This removes *hive.llap.queue.metrics.percentiles.intervals* and
adds *hive.llap.io.decoding.metrics.percentiles.intervals* so the changes need
to be documented in the LLAP section of Configuration Properties for release
2.1.0.
* [Configuration Properties -- LLAP |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP]
Question: Why is *hive.llap.queue.metrics.percentiles.intervals* included at
the end of the description for
*hive.llap.io.decoding.metrics.percentiles.intervals* (or are "\n" and "+" on
the previous line errors)?
{code}
-
LLAP_QUEUE_METRICS_PERCENTILE_INTERVALS("hive.llap.queue.metrics.percentiles.intervals",
"",
+
LLAP_IO_DECODING_METRICS_PERCENTILE_INTERVALS("hive.llap.io.decoding.metrics.percentiles.intervals",
"30",
"Comma-delimited set of integers denoting the desired rollover
intervals (in seconds)\n" +
-"for percentile latency metrics on the LLAP daemon producer-consumer
queue.\n" +
-"By default, percentile latency metrics are disabled."),
+"for percentile latency metrics on the LLAP daemon IO decoding
time.\n" +
+"hive.llap.queue.metrics.percentiles.intervals"),
{code}
> LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
> -
>
> Key: HIVE-13494
> URL: https://issues.apache.org/jira/browse/HIVE-13494
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch,
> HIVE-13494.2.patch
>
>
> LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics.
> Async IO metrics also seems incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13494: -- Labels: TODOC2.1 (was: ) > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13536: - Attachment: HIVE-13536.2.patch Addressed [~sseth]'s review comments. > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13536) LLAP: Add metrics for task scheduler
[ https://issues.apache.org/jira/browse/HIVE-13536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13536: - Status: Patch Available (was: Open) > LLAP: Add metrics for task scheduler > > > Key: HIVE-13536 > URL: https://issues.apache.org/jira/browse/HIVE-13536 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13536.1.patch, HIVE-13536.2.patch > > > Currently there are no metrics for task scheduler. It will be useful to > provide one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.1.0 (was: 2.1.0, 2.0.1) Status: Resolved (was: Patch Available) Committed to master. Thanks [~sseth] for the review! > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255524#comment-15255524 ] Prasanth Jayachandran commented on HIVE-13494: -- These test failure does not look related. I ran all the llap test on local machine and all of them passed. It's probably some intermittent issue that caused these test to fails on precommit. > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255520#comment-15255520 ] Lefty Leverenz commented on HIVE-13480: --- Doc note: This adds *hive.service.metrics.hadoop2.frequency* and *hive.service.metrics.hadoop2.component* to HiveConf.java and adds HADOOP2 to the description of *hive.service.metrics.reporter*. They need to be documented in the Metrics section of Configuration Properties for release 2.1.0. * [Configuration Properties -- Metrics | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Metrics] > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13494) LLAP: Some metrics from daemon are not exposed to hadoop-metrics2
[ https://issues.apache.org/jira/browse/HIVE-13494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13494: - Attachment: (was: HIVE-13494.2.patch) > LLAP: Some metrics from daemon are not exposed to hadoop-metrics2 > - > > Key: HIVE-13494 > URL: https://issues.apache.org/jira/browse/HIVE-13494 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0, 2.0.1 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13494.1.patch, HIVE-13494.1.patch, > HIVE-13494.2.patch > > > LlapDaemonInfo is exposed via JMX but not sent to hadoop metrics. > Async IO metrics also seems incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13480: -- Labels: TODOC2.1 (was: ) > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
