[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798728#comment-16798728
]
Hive QA commented on HIVE-21386:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963351/HIVE-21386.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=230)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
(batchId=265)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16627/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16627/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16627/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963351 - PreCommit-HIVE-Build
> Extend the fetch task enhancement done in HIVE-21279 to make it work with
> query result cache
>
>
> Key: HIVE-21386
> URL: https://issues.apache.org/jira/browse/HIVE-21386
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch,
> HIVE-21386.3.patch
>
>
> The improvement done in HIVE-21279 is disabled for query cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798714#comment-16798714
]
Hive QA commented on HIVE-21386:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 30 new + 755 unchanged - 2
fixed = 785 total (was 757) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
23s{color} | {color:red} ql generated 1 new + 2254 unchanged - 1 fixed = 2255
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to fakePathExists in
org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.setEntryValid(QueryResultsCache$CacheEntry,
FetchWork) At
QueryResultsCache.java:org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.setEntryValid(QueryResultsCache$CacheEntry,
FetchWork) At QueryResultsCache.java:[line 551] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16627/dev-support/hive-personality.sh
|
| git revision | master / 2b181dc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16627/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16627/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16627/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Extend the fetch task enhancement done in HIVE-21279 to make it work with
> query result cache
>
>
> Key: HIVE-21386
> URL: https://issues.apache.org/jira/browse/HIVE-21386
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch,
> HIVE-21386.3.patch
>
>
> The improvement done in HIVE-21279 is disabled for query cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21283: Component/s: UDF > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature > Components: UDF >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.10.PATCH, > HIVE.21283.2.PATCH, HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, > image-2019-03-16-21-33-18-898.png > > Time Spent: 3h > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21283: Resolution: Fixed Status: Resolved (was: Patch Available) 10.patch committed to master. Thanks [~rmsm...@gmail.com] for the patch! > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.10.PATCH, > HIVE.21283.2.PATCH, HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, > image-2019-03-16-21-33-18-898.png > > Time Spent: 3h > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21283) Create Synonym mid for substr, position for locate
[ https://issues.apache.org/jira/browse/HIVE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798697#comment-16798697 ] Sankar Hariappan commented on HIVE-21283: - +1 > Create Synonym mid for substr, position for locate > > > Key: HIVE-21283 > URL: https://issues.apache.org/jira/browse/HIVE-21283 > Project: Hive > Issue Type: New Feature >Reporter: Mani M >Assignee: Mani M >Priority: Minor > Labels: UDF, pull-request-available, todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE.21283.03.PATCH, HIVE.21283.04.PATCH, > HIVE.21283.05.PATCH, HIVE.21283.06.PATCH, HIVE.21283.07.PATCH, > HIVE.21283.08.PATCH, HIVE.21283.09.PATCH, HIVE.21283.10.PATCH, > HIVE.21283.2.PATCH, HIVE.21283.PATCH, image-2019-03-16-21-31-15-541.png, > image-2019-03-16-21-33-18-898.png > > Time Spent: 3h > Remaining Estimate: 0h > > Create new synonym for the existing function > > Mid for substr > postiion for locate -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798685#comment-16798685
]
Hive QA commented on HIVE-21469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963345/HIVE-21469.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 918 failed/errored test(s), 15360 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=101)
[udf_array_contains_wrong2.q,alter_partition_coltype_invalidcolname.q,authorization_role_cycles2.q,archive_corrupt.q,ptf_window_boundaries2.q,merge_negative_3.q,clusterbysortby.q,fallbackauth_create_func2.q,clustern3.q,invalid_select_expression.q,subquery_select_aggregate.q,authorization_ctas2.q,dynamic_partitions_with_whitelist.q,authorization_public_create.q,set_hiveconf_validation2.q,create_or_replace_view5.q,invalid_stddev_samp_syntax.q,select_udtf_alias.q,windowing_leadlag_in_udaf.q,authorization_truncate.q,analyze_view.q,exim_14_nonpart_part.q,orc_change_fileformat.q,delete_non_acid_table.q,create_with_constraints_validate.q,orc_replace_columns1_acid.q,describe_xpath3.q,input4_2.q,archive_multi1.q,orc_replace_columns2_acid.q,invalid_varchar_length_3.q,authorization_create_tbl.q,add_partition_with_whitelist.q,check_constraint_qual_name.q,authorization_uri_altertab_setloc.q,groupby_grouping_id1.q,create_udaf_failure.q,invalid_t_alter1.q,subquery_select_distinct.q,udf_size_wrong_type.q,exim_04_nonpart_noncompat_colnumber.q,archive_partspec2.q,join2.q,exim_03_nonpart_noncompat_colschema.q,input2.q,authorization_fail_6.q,ptf_negative_WhereWithRankCond.q,dyn_part4.q,spark_task_failure.q,authorization_uri_insert_local.q,materialized_view_no_supported_op_rewrite_2.q,drop_database_cascade.q,duplicate_alias_in_transform.q,alter_view_failure5.q,subquery_corr_select.q,partition_column_names_with_leading_and_trailing_spaces.q,view_delete.q,load_orc_negative3.q,alter_table_wrong_table.q,subquery_scalar_multi_rows.q,authorization_create_role_no_admin.q,exim_00_unsupported_schema.q,semijoin4.q,compile_processor.q,create_unknown_genericudf.q,udf_function_does_not_implement_udf.q,orc_type_promotion3_acid.q,udf_local_resource.q,merge_negative_4.q,lockneg_query_tbl_in_locked_db.q,udaf_collect_set_unsupported.q,update_non_acid_table.q,insert_into2.q,exchange_partition_neg_partition_exists3.q,join28.q,alter_view_failure7.q,invalid_cast_to_binary_1.q,change_hive_tmp_table_space.q,fileformat_void_output.q,wrong_column_type.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=99)
[jira] [Work logged] (HIVE-21485) Hive desc operation takes more than 100 seconds after upgrading from Hive 1.2.1 to 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21485?focusedWorklogId=217109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-217109
]
ASF GitHub Bot logged work on HIVE-21485:
-
Author: ASF GitHub Bot
Created on: 22/Mar/19 03:17
Start Date: 22/Mar/19 03:17
Worklog Time Spent: 10m
Work Description: rmsmani commented on issue #577: HIVE-21485: Add flag
to turn off fetching partition stats in DESCRIBE…
URL: https://github.com/apache/hive/pull/577#issuecomment-475480051
1. Create a patch file (patch file is diff between the latest commit as
available in master with your proposed changes)
2. Upload the patch file to Jira card
3 click the submit patch button
The automatic pre built testing will be triggered at Jenkins and the results
will be uploaded to Jira card upon build completion.
For more details check how to contribute to hive wiki
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 217109)
Time Spent: 20m (was: 10m)
> Hive desc operation takes more than 100 seconds after upgrading from Hive
> 1.2.1 to 2.3.4
>
>
> Key: HIVE-21485
> URL: https://issues.apache.org/jira/browse/HIVE-21485
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive
>Affects Versions: 2.3.4
>Reporter: Qingxin Wu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hive desc [formatted|extended] operation cost more than 100 seconds after
> upgrading from Hive 1.2.1 to 2.3.4. This is mainly caused by showing stats
> for partitioned tables which was introduced by HIVE-16098 when the
> partitioned tables have a large amount of partitions. In our case, the number
> of partition is 187221.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 115.342 seconds, Fetched: 42 row(s)
> {code}
> same operation executed in hive-1.2.1 and only cost 2 seconds.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 2.037 seconds, Fetched: 42 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798661#comment-16798661
]
Hive QA commented on HIVE-21469:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 49 unchanged - 34
fixed = 50 total (was 83) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
32s{color} | {color:red} ql generated 1 new + 2252 unchanged - 3 fixed = 2253
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Write to static field
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.metrics
from instance method
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(HiveLockManagerCtx)
At ZooKeeperHiveLockManager.java:from instance method
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(HiveLockManagerCtx)
At ZooKeeperHiveLockManager.java:[line 124] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16626/dev-support/hive-personality.sh
|
| git revision | master / b953c4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16626/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16626/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16626/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch, HIVE-21469.4.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new
[jira] [Updated] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[ https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21386: --- Status: Open (was: Patch Available) > Extend the fetch task enhancement done in HIVE-21279 to make it work with > query result cache > > > Key: HIVE-21386 > URL: https://issues.apache.org/jira/browse/HIVE-21386 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch, > HIVE-21386.3.patch > > > The improvement done in HIVE-21279 is disabled for query cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[ https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21386: --- Status: Patch Available (was: Open) > Extend the fetch task enhancement done in HIVE-21279 to make it work with > query result cache > > > Key: HIVE-21386 > URL: https://issues.apache.org/jira/browse/HIVE-21386 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch, > HIVE-21386.3.patch > > > The improvement done in HIVE-21279 is disabled for query cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[ https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21386: --- Attachment: HIVE-21386.3.patch > Extend the fetch task enhancement done in HIVE-21279 to make it work with > query result cache > > > Key: HIVE-21386 > URL: https://issues.apache.org/jira/browse/HIVE-21386 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch, > HIVE-21386.3.patch > > > The improvement done in HIVE-21279 is disabled for query cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798642#comment-16798642
]
Hive QA commented on HIVE-21204:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963339/HIVE-21204.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15836 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.metrics.TestReadWriteLockMetrics.testWithContention
(batchId=330)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16625/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16625/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16625/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963339 - PreCommit-HIVE-Build
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.4.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798619#comment-16798619
]
Hive QA commented on HIVE-21204:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} llap-common in master has 76 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 79 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
22s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s{color}
| {color:red} llap-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} llap-common: The patch generated 20 new + 0 unchanged
- 0 fixed = 20 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 5 new + 146 unchanged
- 4 fixed = 151 total (was 150) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red} llap-common generated 2 new + 76 unchanged - 0 fixed =
78 total (was 76) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-common |
| | Should
org.apache.hadoop.hive.llap.metrics.ReadWriteLockMetrics$LockWrapper be a
_static_ inner class? At ReadWriteLockMetrics.java:inner class? At
ReadWriteLockMetrics.java:[lines 335-394] |
| |
org.apache.hadoop.hive.llap.metrics.ReadWriteLockMetrics$MetricsComparator
implements Comparator but not Serializable At
ReadWriteLockMetrics.java:Serializable At ReadWriteLockMetrics.java:[lines
62-88] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16625/dev-support/hive-personality.sh
|
| git revision | master / b953c4e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16625/yetus/patch-mvninstall-llap-server.txt
|
| compile |
[jira] [Updated] (HIVE-21485) Hive desc operation takes more than 100 seconds after upgrading from Hive 1.2.1 to 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21485:
--
Labels: pull-request-available (was: )
> Hive desc operation takes more than 100 seconds after upgrading from Hive
> 1.2.1 to 2.3.4
>
>
> Key: HIVE-21485
> URL: https://issues.apache.org/jira/browse/HIVE-21485
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive
>Affects Versions: 2.3.4
>Reporter: Qingxin Wu
>Priority: Major
> Labels: pull-request-available
>
> Hive desc [formatted|extended] operation cost more than 100 seconds after
> upgrading from Hive 1.2.1 to 2.3.4. This is mainly caused by showing stats
> for partitioned tables which was introduced by HIVE-16098 when the
> partitioned tables have a large amount of partitions. In our case, the number
> of partition is 187221.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 115.342 seconds, Fetched: 42 row(s)
> {code}
> same operation executed in hive-1.2.1 and only cost 2 seconds.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 2.037 seconds, Fetched: 42 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21485) Hive desc operation takes more than 100 seconds after upgrading from Hive 1.2.1 to 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21485?focusedWorklogId=217089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-217089
]
ASF GitHub Bot logged work on HIVE-21485:
-
Author: ASF GitHub Bot
Created on: 22/Mar/19 01:39
Start Date: 22/Mar/19 01:39
Worklog Time Spent: 10m
Work Description: TopGunViper commented on pull request #577: HIVE-21485:
Add flag to turn off fetching partition stats in DESCRIBE…
URL: https://github.com/apache/hive/pull/577
## What changes were proposed in this pull request?
Hive DESCRIBE [formatted|extended] operation cost more than 100 seconds
after upgrading from Hive 1.2.1 to 2.3.4. This is mainly caused by showing
stats for partitioned tables which was introduced by
[HIVE-16098](https://issues.apache.org/jira/browse/HIVE-16098) when the
partitioned tables have a large amount of partitions.
So，could we add a flag that determines whether 'DESCRIBE
[EXTENDED|FORMATTED]' operation display partitioned table stats or not?
## How was this patch tested?
Query Unit Test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 217089)
Time Spent: 10m
Remaining Estimate: 0h
> Hive desc operation takes more than 100 seconds after upgrading from Hive
> 1.2.1 to 2.3.4
>
>
> Key: HIVE-21485
> URL: https://issues.apache.org/jira/browse/HIVE-21485
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive
>Affects Versions: 2.3.4
>Reporter: Qingxin Wu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive desc [formatted|extended] operation cost more than 100 seconds after
> upgrading from Hive 1.2.1 to 2.3.4. This is mainly caused by showing stats
> for partitioned tables which was introduced by HIVE-16098 when the
> partitioned tables have a large amount of partitions. In our case, the number
> of partition is 187221.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 115.342 seconds, Fetched: 42 row(s)
> {code}
> same operation executed in hive-1.2.1 and only cost 2 seconds.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 2.037 seconds, Fetched: 42 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Status: Patch Available (was: Open)
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch, HIVE-21469.4.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked. Also, when the {{InterruptedException}}
> is caught in the try block, the thread's interrupted flag is cleared. The
> flag is not reset in this code and therefore we lose the fact that this
> thread has been interrupted. This flag should be preserved so that other
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Attachment: HIVE-21469.4.patch
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch, HIVE-21469.4.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked. Also, when the {{InterruptedException}}
> is caught in the try block, the thread's interrupted flag is cleared. The
> flag is not reset in this code and therefore we lose the fact that this
> thread has been interrupted. This flag should be preserved so that other
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Status: Open (was: Patch Available)
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked. Also, when the {{InterruptedException}}
> is caught in the try block, the thread's interrupted flag is cleared. The
> flag is not reset in this code and therefore we lose the fact that this
> thread has been interrupted. This flag should be preserved so that other
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[
https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798600#comment-16798600
]
Hive QA commented on HIVE-21183:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963323/HIVE-21183.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerDagTotalTasks
(batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16624/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16624/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16624/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963323 - PreCommit-HIVE-Build
> Interrupt wait time for FileCacheCleanupThread
> --
>
> Key: HIVE-21183
> URL: https://issues.apache.org/jira/browse/HIVE-21183
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Minor
> Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch,
> HIVE-21183.2.patch, HIVE-21183.patch
>
>
> The FileCacheCleanupThread is waiting unnecessarily long for eviction counts
> to increment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21485) Hive desc operation takes more than 100 seconds after upgrading from Hive 1.2.1 to 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798594#comment-16798594
]
star edited comment on HIVE-21485 at 3/22/19 12:49 AM:
---
Maybe table with so much partitions is not a good practice.
was (Author: starphin):
Maybe table with so much partition is not a good practice.
> Hive desc operation takes more than 100 seconds after upgrading from Hive
> 1.2.1 to 2.3.4
>
>
> Key: HIVE-21485
> URL: https://issues.apache.org/jira/browse/HIVE-21485
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive
>Affects Versions: 2.3.4
>Reporter: Qingxin Wu
>Priority: Major
>
> Hive desc [formatted|extended] operation cost more than 100 seconds after
> upgrading from Hive 1.2.1 to 2.3.4. This is mainly caused by showing stats
> for partitioned tables which was introduced by HIVE-16098 when the
> partitioned tables have a large amount of partitions. In our case, the number
> of partition is 187221.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 115.342 seconds, Fetched: 42 row(s)
> {code}
> same operation executed in hive-1.2.1 and only cost 2 seconds.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 2.037 seconds, Fetched: 42 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21485) Hive desc operation takes more than 100 seconds after upgrading from Hive 1.2.1 to 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798594#comment-16798594
]
star commented on HIVE-21485:
-
Maybe table with so much partition is not a good practice.
> Hive desc operation takes more than 100 seconds after upgrading from Hive
> 1.2.1 to 2.3.4
>
>
> Key: HIVE-21485
> URL: https://issues.apache.org/jira/browse/HIVE-21485
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive
>Affects Versions: 2.3.4
>Reporter: Qingxin Wu
>Priority: Major
>
> Hive desc [formatted|extended] operation cost more than 100 seconds after
> upgrading from Hive 1.2.1 to 2.3.4. This is mainly caused by showing stats
> for partitioned tables which was introduced by HIVE-16098 when the
> partitioned tables have a large amount of partitions. In our case, the number
> of partition is 187221.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 115.342 seconds, Fetched: 42 row(s)
> {code}
> same operation executed in hive-1.2.1 and only cost 2 seconds.
> {code:java}
> hive> desc bus.kafka_data;
> OK
> idstring
> ...
> d map
> stat_date string
> log_idstring
> # Partition Information
> # col_namedata_type comment
> stat_date string
> log_idstring
> Time taken: 2.037 seconds, Fetched: 42 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables
[ https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21482: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Thanks Jason for the review. Committed to master. > Partition discovery table property is added to non-partitioned external tables > -- > > Key: HIVE-21482 > URL: https://issues.apache.org/jira/browse/HIVE-21482 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21482.1.patch, HIVE-21482.2.patch > > > Automatic partition discovery is added to external tables by default. But it > doesn't check if the external table is partitioned or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oliver Draese updated HIVE-21204:
-
Attachment: (was: HIVE-21204.2.patch)
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.4.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oliver Draese updated HIVE-21204:
-
Attachment: (was: HIVE-21204.1.patch)
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.4.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oliver Draese updated HIVE-21204:
-
Attachment: (was: HIVE-21204.patch)
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.4.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oliver Draese updated HIVE-21204:
-
Attachment: (was: HIVE-21204.3.patch)
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.4.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oliver Draese updated HIVE-21204:
-
Attachment: HIVE-21204.4.patch
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.4.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[
https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798576#comment-16798576
]
Hive QA commented on HIVE-21183:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 79 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red} llap-server generated 1 new + 79 unchanged - 0 fixed =
80 total (was 79) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-server |
| | Synchronization performed on java.util.concurrent.atomic.AtomicInteger in
org.apache.hadoop.hive.llap.cache.SerDeLowLevelCacheImpl.notifyEvicted(MemoryBuffer)
At
SerDeLowLevelCacheImpl.java:org.apache.hadoop.hive.llap.cache.SerDeLowLevelCacheImpl.notifyEvicted(MemoryBuffer)
At SerDeLowLevelCacheImpl.java:[line 666] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16624/dev-support/hive-personality.sh
|
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16624/yetus/new-findbugs-llap-server.html
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16624/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Interrupt wait time for FileCacheCleanupThread
> --
>
> Key: HIVE-21183
> URL: https://issues.apache.org/jira/browse/HIVE-21183
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Minor
> Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch,
> HIVE-21183.2.patch, HIVE-21183.patch
>
>
> The FileCacheCleanupThread is waiting unnecessarily long for eviction counts
> to increment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798564#comment-16798564
]
Hive QA commented on HIVE-21204:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963321/HIVE-21204.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16623/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16623/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16623/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:53:35.397
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16623/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:53:35.401
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:53:36.457
+ rm -rf ../yetus_PreCommit-HIVE-Build-16623
+ mkdir ../yetus_PreCommit-HIVE-Build-16623
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16623
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16623/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java:
does not exist in index
error:
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc6516389071222815358.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc6516389071222815358.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc3890435263173508293.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Commented] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798559#comment-16798559
]
Hive QA commented on HIVE-21469:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963319/HIVE-21469.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16622/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16622/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16622/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:49:30.153
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16622/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:49:30.155
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:49:31.217
+ rm -rf ../yetus_PreCommit-HIVE-Build-16622
+ mkdir ../yetus_PreCommit-HIVE-Build-16622
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16622
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16622/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockMode.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObj.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2085600430980394434.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2085600430980394434.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc5705060335407942899.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798557#comment-16798557
]
Hive QA commented on HIVE-21422:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963317/HIVE-21422.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16621/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16621/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16621/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:47:40.624
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16621/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:47:40.628
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 38682a4 HIVE-21446 : Hive Server going OOM during hive external
table replications. (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-21 23:47:41.326
+ rm -rf ../yetus_PreCommit-HIVE-Build-16621
+ mkdir ../yetus_PreCommit-HIVE-Build-16621
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16621
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16621/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc5208815536948858716.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc5208815536948858716.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16621
+ exit 1
'
{noformat}
This message is automatically
[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798554#comment-16798554
]
Hive QA commented on HIVE-9995:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16619/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16619/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16619/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963282 - PreCommit-HIVE-Build
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798555#comment-16798555
]
Hive QA commented on HIVE-9995:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16620/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16620/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16620/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963282 - PreCommit-HIVE-Build
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798552#comment-16798552
]
Hive QA commented on HIVE-21465:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963272/HIVE-21465.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16618/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16618/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16618/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963272 - PreCommit-HIVE-Build
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch, HIVE-21465.02.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798536#comment-16798536
]
Hive QA commented on HIVE-21465:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
22s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 3 new + 9 unchanged - 23 fixed
= 12 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16618/dev-support/hive-personality.sh
|
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16618/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16618/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16618/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch, HIVE-21465.02.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21409) Initial SessionState ClassLoader Reused For Subsequent Sessions
[ https://issues.apache.org/jira/browse/HIVE-21409?focusedWorklogId=217061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-217061 ] ASF GitHub Bot logged work on HIVE-21409: - Author: ASF GitHub Bot Created on: 21/Mar/19 22:57 Start Date: 21/Mar/19 22:57 Worklog Time Spent: 10m Work Description: shawnweeks commented on issue #575: HIVE-21409 Add Jars to Session Conf ClassLoader URL: https://github.com/apache/hive/pull/575#issuecomment-475433844 Closing this as this breaks Ranger. Will continue troubleshooting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 217061) Time Spent: 0.5h (was: 20m) > Initial SessionState ClassLoader Reused For Subsequent Sessions > --- > > Key: HIVE-21409 > URL: https://issues.apache.org/jira/browse/HIVE-21409 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Shawn Weeks >Priority: Minor > Labels: pull-request-available > Attachments: create_class.sql, run.sql, setup.sql > > Time Spent: 0.5h > Remaining Estimate: 0h > > It appears that the first ClassLoader attached to a SessionState Static > Instance is being reused as the parent for all future sessions. This causes > any libraries added to the class path on the initial session to be added to > future sessions. It also appears that further sessions may be adding jars to > this initial ClassLoader as well leading to the class path getting more and > more polluted. This occurring on a build including HIVE-11878. I've included > some examples that greatly exaggerate the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21409) Initial SessionState ClassLoader Reused For Subsequent Sessions
[ https://issues.apache.org/jira/browse/HIVE-21409?focusedWorklogId=217062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-217062 ] ASF GitHub Bot logged work on HIVE-21409: - Author: ASF GitHub Bot Created on: 21/Mar/19 22:57 Start Date: 21/Mar/19 22:57 Worklog Time Spent: 10m Work Description: shawnweeks commented on pull request #575: HIVE-21409 Add Jars to Session Conf ClassLoader URL: https://github.com/apache/hive/pull/575 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 217062) Time Spent: 40m (was: 0.5h) > Initial SessionState ClassLoader Reused For Subsequent Sessions > --- > > Key: HIVE-21409 > URL: https://issues.apache.org/jira/browse/HIVE-21409 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Shawn Weeks >Priority: Minor > Labels: pull-request-available > Attachments: create_class.sql, run.sql, setup.sql > > Time Spent: 40m > Remaining Estimate: 0h > > It appears that the first ClassLoader attached to a SessionState Static > Instance is being reused as the parent for all future sessions. This causes > any libraries added to the class path on the initial session to be added to > future sessions. It also appears that further sessions may be adding jars to > this initial ClassLoader as well leading to the class path getting more and > more polluted. This occurring on a build including HIVE-11878. I've included > some examples that greatly exaggerate the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[
https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798526#comment-16798526
]
Hive QA commented on HIVE-21467:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} hcatalog/core in master has 28 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 11s{color}
| {color:red} storage-api generated 2 new + 0 unchanged - 2 fixed = 2 total
(was 2) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
44s{color} | {color:red} patch/serde cannot
[jira] [Commented] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[
https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798497#comment-16798497
]
Hive QA commented on HIVE-21467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963267/HIVE-21467.02.patch
{color:green}SUCCESS:{color} +1 due to 96 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16617/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16617/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16617/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963267 - PreCommit-HIVE-Build
> Remove deprecated junit.framework.Assert imports
>
>
> Key: HIVE-21467
> URL: https://issues.apache.org/jira/browse/HIVE-21467
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Labels: newbie
> Attachments: HIVE-21467.01.patch, HIVE-21467.02.patch,
> HIVE-21467.02.patch
>
>
> These imports trigger lots of warnings in ide, which could be annoying, and
> it can be replaced easily to org.junit.Assert, the signature and behavior are
> the same, so the tests should pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.2.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, > HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oliver Draese updated HIVE-21204:
-
Attachment: HIVE-21204.3.patch
> Instrumentation for read/write locks in LLAP
>
>
> Key: HIVE-21204
> URL: https://issues.apache.org/jira/browse/HIVE-21204
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Attachments: HIVE-21204.1.patch, HIVE-21204.2.patch,
> HIVE-21204.3.patch, HIVE-21204.patch
>
>
> LLAP has several R/W locks for serialization of updates to query tracker,
> file data,
> Instrumentation is added to monitor the
> * total amount of R/W locks within a particular category
> * average + max wait/suspension time to get the R/W lock
> A category includes all lock instances for particular areas (i.e. category is
> FileData and all R/W locks that are used in FileData instances are accounted
> within the one category).
> The monitoring/accounting is done via Hadoop Metrics 2, making them
> accessible via JMX. In addition, a new "locking" GET endpoint is added to the
> LLAP daemon's REST interface. It produces output like the following example:
> {
> {{ "statsCollection": "enabled",}}
> {{ "lockStats": [}}
> {{ {}}{{ "type": "R/W Lock Stats",}}
> {{ "label": "FileData",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ },}}
> {{ { "}}{{type": "R/W Lock Stats",}}
> {{ "label": "QueryTracker",}}
> {{ "totalLockWaitTimeMillis": 0,}}
> {{ "readLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> {{ },}}
> {{ "writeLock": {}}
> {{ "count": 0,}}
> {{ "avgWaitTimeNanos": 0,}}
> {{ "maxWaitTimeNanos": 0}}
> }
> {{ } }}{{]}}
> {{}}}
> To avoid the overhead of lock instrumentation, lock metrics collection is
> disabled by default and can be enabled via the following configuration
> parameter:
> {{hive.llap.lockmetrics.collect = true}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Attachment: HIVE-21469.3.patch
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked. Also, when the {{InterruptedException}}
> is caught in the try block, the thread's interrupted flag is cleared. The
> flag is not reset in this code and therefore we lose the fact that this
> thread has been interrupted. This flag should be preserved so that other
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Status: Open (was: Patch Available)
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked. Also, when the {{InterruptedException}}
> is caught in the try block, the thread's interrupted flag is cleared. The
> flag is not reset in this code and therefore we lose the fact that this
> thread has been interrupted. This flag should be preserved so that other
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21469) Review of ZooKeeperHiveLockManager
[
https://issues.apache.org/jira/browse/HIVE-21469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21469:
--
Status: Patch Available (was: Open)
> Review of ZooKeeperHiveLockManager
> --
>
> Key: HIVE-21469
> URL: https://issues.apache.org/jira/browse/HIVE-21469
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21469.1.patch, HIVE-21469.2.patch,
> HIVE-21469.3.patch
>
>
> A lot of sins in this class to resolve:
> {code:java}
> @Override
> public void setContext(HiveLockManagerCtx ctx) throws LockException {
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
> LOG.warn("Unexpected ZK exception when creating parent node /" +
> parent, e);
> }
> }
> {code}
> Every time a new session is created and this {{setContext}} method is called,
> it attempts to create the root node. I have seen that, even though the root
> node exists, an create node action is written to the ZK logs. Check first if
> the node exists before trying to create it.
> {code:java}
> try {
> curatorFramework.delete().forPath(zLock.getPath());
> } catch (InterruptedException ie) {
> curatorFramework.delete().forPath(zLock.getPath());
> }
> {code}
> There has historically been a quite a few bugs regarding leaked locks. The
> Driver will signal the session {{Thread}} by performing an interrupt. That
> interrupt can happen any time and it can kill a create/delete action within
> the ZK framework. We can see one example of workaround for this. If the ZK
> action is interrupted, simply do it again. Well, what if it's interrupted
> yet again? The lock will be leaked. Also, when the {{InterruptedException}}
> is caught in the try block, the thread's interrupted flag is cleared. The
> flag is not reset in this code and therefore we lose the fact that this
> thread has been interrupted. This flag should be preserved so that other
> code paths will know that it's time to exit.
> {code:java}
> if (tryNum > 1) {
> Thread.sleep(sleepTime);
> }
> unlockPrimitive(hiveLock, parent, curatorFramework);
> break;
> } catch (Exception e) {
> if (tryNum >= numRetriesForUnLock) {
> String name = ((ZooKeeperHiveLock)hiveLock).getPath();
> throw new LockException("Node " + name + " can not be deleted after
> " + numRetriesForUnLock + " attempts.",
> e);
> }
> }
> {code}
> ... related... the sleep here may be interrupted, but we still need to delete
> the lock (again, for fear of leaking it). This sleep should be
> uninterruptible. If we need to get the lock deleted, and there's a problem,
> interrupting the sleep will cause the code to eventually exit and locks will
> be leaked.
> It also requires a bunch more TLC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.2.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, > HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15406) Consider vectorizing the new 'trunc' function
[
https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798428#comment-16798428
]
Hive QA commented on HIVE-15406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963257/HIVE-15406.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15834 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16616/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16616/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16616/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963257 - PreCommit-HIVE-Build
> Consider vectorizing the new 'trunc' function
> -
>
> Key: HIVE-15406
> URL: https://issues.apache.org/jira/browse/HIVE-15406
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Matt McCline
>Assignee: Laszlo Bodor
>Priority: Critical
> Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch,
> HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch,
> HIVE-15406.06.patch
>
>
> Rounding function 'trunc' added by HIVE-14582.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15406) Consider vectorizing the new 'trunc' function
[
https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798403#comment-16798403
]
Hive QA commented on HIVE-15406:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 16 new + 4 unchanged - 1 fixed
= 20 total (was 5) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 7 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16616/dev-support/hive-personality.sh
|
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16616/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16616/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16616/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16616/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Consider vectorizing the new 'trunc' function
> -
>
> Key: HIVE-15406
> URL: https://issues.apache.org/jira/browse/HIVE-15406
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Matt McCline
>Assignee: Laszlo Bodor
>Priority: Critical
> Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch,
> HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch,
> HIVE-15406.06.patch
>
>
> Rounding function 'trunc' added by HIVE-14582.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798372#comment-16798372
]
Hive QA commented on HIVE-9995:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16615/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16615/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16615/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963282 - PreCommit-HIVE-Build
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798349#comment-16798349
]
Hive QA commented on HIVE-9995:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} ql: The patch generated 0 new + 702 unchanged - 10
fixed = 702 total (was 712) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16615/dev-support/hive-personality.sh
|
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16615/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0
[jira] [Commented] (HIVE-21443) Better usability for SHOW COMPACTIONS
[ https://issues.apache.org/jira/browse/HIVE-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798322#comment-16798322 ] Peter Vary commented on HIVE-21443: --- [~gopalv], [~tlipcon]: I think adding this to the sys.COMPACTIONS could be done adding the new tables here. See: [https://github.com/apache/hive/blob/master/metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql] As a follow-up jira, maybe the SQL scripts creating the schema should be moved to the standalone-metastore directory as well (just to make the review easier). Is it enough to have the same data exposed as for the SHOW COMPACTIONS command?: * CompactionId * Database * Table * Partition * Type * State * Hostname * Worker * Start Time * Duration(ms) * HadoopJobId The missing columns are: * CC_TBLPROPERTIES varchar(2048), * CC_WORKER_ID varchar(128), * CC_RUN_AS varchar(128), * CC_HIGHEST_WRITE_ID bigint, * CC_META_INFO varbinary(2048), If you guys do not have an opinion, I would go for the same data as for the original command. What should we do with the SHOW COMPACTIONS command? Will this expose dbnames/tablenames/partitionnames for unauthorized users? Thanks, Peter > Better usability for SHOW COMPACTIONS > - > > Key: HIVE-21443 > URL: https://issues.apache.org/jira/browse/HIVE-21443 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Todd Lipcon >Priority: Major > > Currently on a test cluster the output of 'SHOW COMPACTIONS' has 117k rows. > This makes it basically useless to work with. > For better usability, we should support syntax like 'SHOW COMPACTIONS IN > ' or maybe 'SHOW COMPACTIONS ON ' (particular syntax to be > chosen for consistency with other operations I suppose). > Alternatively (or maybe in addition) it seems like it would be nice to expose > the same data in a queryable table (eg in information_schema or a system > namespace) so that I could do things like: SELECT dbname, state, count(*) > from compactions group by 1,2; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21473) Bumping jackson version
[
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798318#comment-16798318
]
Hive QA commented on HIVE-21473:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963247/HIVE-21473.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16614/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16614/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16614/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963247 - PreCommit-HIVE-Build
> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.2.patch, HIVE-21473.3.patch, HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21473) Bumping jackson version
[
https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798311#comment-16798311
]
Hive QA commented on HIVE-21473:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16614/dev-support/hive-personality.sh
|
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16614/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore . testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16614/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Bumping jackson version
> ---
>
> Key: HIVE-21473
> URL: https://issues.apache.org/jira/browse/HIVE-21473
> Project: Hive
> Issue Type: Task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21473.2.patch, HIVE-21473.3.patch, HIVE-21473.patch
>
>
> Bump jackson version to 2.9.8
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
[
https://issues.apache.org/jira/browse/HIVE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798278#comment-16798278
]
Hive QA commented on HIVE-21290:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963240/HIVE-21290.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_analyze]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_external_time]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_stats]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=118)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16613/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16613/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16613/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963240 - PreCommit-HIVE-Build
> Restore historical way of handling timestamps in Parquet while keeping the
> new semantics at the same time
> -
>
> Key: HIVE-21290
> URL: https://issues.apache.org/jira/browse/HIVE-21290
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Ivanfi
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21290.1.patch, HIVE-21290.2.patch,
> HIVE-21290.2.patch
>
>
> This sub-task is for implementing the Parquet-specific parts of the following
> plan:
> h1. Problem
> Historically, the semantics of the TIMESTAMP type in Hive depended on the
> file format. Timestamps in Avro, Parquet and RCFiles with a binary SerDe had
> _Instant_ semantics, while timestamps in ORC, textfiles and RCFiles with a
> text SerDe had _LocalDateTime_ semantics.
> The Hive community wanted to get rid of this inconsistency and have
> _LocalDateTime_ semantics in Avro, Parquet and RCFiles with a binary SerDe as
> well. *Hive 3.1 turned off normalization to UTC* to achieve this. While this
> leads to the desired new semantics, it also leads to incorrect results when
> new Hive versions read timestamps written by old Hive versions or when old
> Hive versions or any other component not aware of this change (including
> legacy Impala and Spark versions) read timestamps written by new Hive
> versions.
> h1. Solution
> To work around this issue, Hive *should restore the practice of normalizing
> to UTC* when writing timestamps to Avro, Parquet and RCFiles with a binary
> SerDe. In itself, this would restore the historical _Instant_ semantics,
> which is undesirable. In order to achieve the desired _LocalDateTime_
> semantics in spite of normalizing to UTC, newer Hive versions should record
> the session-local local time zone in the file metadata fields serving
> arbitrary key-value storage purposes.
> When reading back files with this time zone metadata, newer Hive versions (or
> any other new component aware of this extra metadata) can achieve
> _LocalDateTime_ semantics by *converting from UTC to the saved time zone
> (instead of to the local time zone)*. Legacy components that are unaware of
> the new metadata can read the files without any problem and the timestamps
> will show the historical Instant behaviour to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
[
https://issues.apache.org/jira/browse/HIVE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798277#comment-16798277
]
Hive QA commented on HIVE-21290:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 25 new + 193 unchanged - 22
fixed = 218 total (was 215) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
2s{color} | {color:red} root: The patch generated 25 new + 194 unchanged - 22
fixed = 219 total (was 216) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16613/dev-support/hive-personality.sh
|
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16613/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16613/yetus/diff-checkstyle-root.txt
|
| modules | C: common ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16613/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Restore historical way of handling timestamps in Parquet while keeping the
> new semantics at the same time
> -
>
> Key: HIVE-21290
> URL: https://issues.apache.org/jira/browse/HIVE-21290
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Ivanfi
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21290.1.patch, HIVE-21290.2.patch,
> HIVE-21290.2.patch
>
>
> This sub-task is for implementing the Parquet-specific parts of the following
> plan:
> h1. Problem
> Historically, the semantics of the TIMESTAMP type in Hive depended on the
> file format. Timestamps in Avro, Parquet and
[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
[
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798263#comment-16798263
]
Peter Vary commented on HIVE-20580:
---
[~ashutoshc]: isOriginal(Footer) is called from
{{org.apache.hadoop.hive.llap.io.metadata.OrcFileMetadata}} consturcor to set
the {{isOriginalFormat}} attribute, which in turn is used for the
implementation of the {{org.apache.orc.FileMetadata.isOriginalFormat()}} method:
{code:java}
public final class OrcFileMetadata implements FileMetadata,
ConsumerFileMetadata {
[..]
public OrcFileMetadata(Object fileKey, OrcProto.Footer footer,
OrcProto.PostScript ps,
List stats, List stripes, final
OrcFile.Version fileVersion) {
[..]
this.isOriginalFormat = OrcInputFormat.isOriginal(footer);
[..]
}
[..]
@Override
public boolean isOriginalFormat() {
return isOriginalFormat;
}
[..]
}{code}
Shall the {{OrcFileMetadata.isOriginalFormat()}} method throw an
\{{java.lang.UnsupportedOperationException}} instead?
Thanks,
Peter
> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch,
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking
> for presence of {{hive.acid.key.index}} in the footer. This is only created
> when the file is written by {{OrcRecordUpdater}}. It should instead check
> for presence of Acid metadata columns so that a file can be produced by
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1
> type of event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798250#comment-16798250 ] mahesh kumar behera commented on HIVE-21446: [^HIVE-21446.03.patch] merged to master ..thanks [~sankarh] for review > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch, > HIVE-21446.03.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798224#comment-16798224
]
Todd Lipcon commented on HIVE-21487:
Seems that this cluster is in some state where all compactions are failing (I
think due to some other testing going on with a strange setup). The query is
likely coming from CompactionTxnHandler.checkFailedCompactions.
It's possible this isn't a hot enough code path to be worth a really large
index, but perhaps it's worth one at least on ethe database/table level.
> COMPLETED_COMPACTIONS table missing appropriate indexes
> ---
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Priority: Major
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases
[
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798220#comment-16798220
]
Hive QA commented on HIVE-21034:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963237/HIVE-21034.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15837 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16612/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16612/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16612/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963237 - PreCommit-HIVE-Build
> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch,
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch,
> HIVE-21034.5.patch, HIVE-21034.5.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all
> databases with CASCADE* to remove all data of managed tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases
[
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798194#comment-16798194
]
Ashutosh Chauhan commented on HIVE-21034:
-
+1
> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch,
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch,
> HIVE-21034.5.patch, HIVE-21034.5.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all
> databases with CASCADE* to remove all data of managed tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
[
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798181#comment-16798181
]
Ashutosh Chauhan commented on HIVE-20580:
-
[~pvary] I will get rid of isOriginal(Footer). I don't see it being part of a
public interface and I would rather not leave a public method which is unused
in code.
LGTM otherwise.
> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch,
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking
> for presence of {{hive.acid.key.index}} in the footer. This is only created
> when the file is written by {{OrcRecordUpdater}}. It should instead check
> for presence of Acid metadata columns so that a file can be produced by
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1
> type of event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-9995:
-
Attachment: (was: HIVE-9995.04.patch)
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-9995:
-
Attachment: HIVE-9995.04.patch
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-9995:
-
Attachment: HIVE-9995.04.patch
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-9995:
-
Attachment: (was: HIVE-9995.04.patch)
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-9995:
-
Attachment: HIVE-9995.04.patch
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21034) Add option to schematool to drop Hive databases
[
https://issues.apache.org/jira/browse/HIVE-21034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798159#comment-16798159
]
Hive QA commented on HIVE-21034:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16612/dev-support/hive-personality.sh
|
| git revision | master / 25b14be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16612/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add option to schematool to drop Hive databases
> ---
>
> Key: HIVE-21034
> URL: https://issues.apache.org/jira/browse/HIVE-21034
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Attachments: HIVE-21034.1.patch, HIVE-21034.2.patch,
> HIVE-21034.2.patch, HIVE-21034.3.patch, HIVE-21034.4.patch,
> HIVE-21034.5.patch, HIVE-21034.5.patch, HIVE-21034.5.patch
>
>
> An option to remove all Hive managed data could be a useful addition to
> {{schematool}}.
> I propose to introduce a new flag {{-dropAllDatabases}} that would *drop all
> databases with CASCADE* to remove all data of managed tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21480) Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup
[
https://issues.apache.org/jira/browse/HIVE-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798139#comment-16798139
]
Hive QA commented on HIVE-21480:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963214/HIVE-21480.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16611/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16611/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16611/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12963214/HIVE-21480.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963214 - PreCommit-HIVE-Build
> Fix test TestHiveMetaStore.testJDOPersistanceManagerCleanup
> ---
>
> Key: HIVE-21480
> URL: https://issues.apache.org/jira/browse/HIVE-21480
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Major
> Attachments: HIVE-21480.1.patch, HIVE-21480.patch
>
>
> [TestHiveMetaStore#testJDOPersistanceManagerCleanup|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java#L3140-L3162]
> tests whether the JDO persistence manager cache cleanup was performed
> correctly when a HiveMetaStoreClient executes an API call, and closes. It
> does this by ensuring that the cache object count before the API call, and
> after closing, are the same. However, there are some assumptions that are not
> always correct, and can cause flakiness.
> For example, lingering resources could be present from previous tests or from
> setup depending on how PTest runs it, and can cause the object count to
> sometimes be incorrect. We should rewrite this test to account for this
> flakiness that can occur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798138#comment-16798138
]
Hive QA commented on HIVE-21386:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963209/HIVE-21386.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1
(batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16610/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16610/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16610/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963209 - PreCommit-HIVE-Build
> Extend the fetch task enhancement done in HIVE-21279 to make it work with
> query result cache
>
>
> Key: HIVE-21386
> URL: https://issues.apache.org/jira/browse/HIVE-21386
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch
>
>
> The improvement done in HIVE-21279 is disabled for query cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: HIVE-21465.02.patch
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch, HIVE-21465.02.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21486) FinalSelectOps is empty in lineage index if there is a script operator(transform)
[ https://issues.apache.org/jira/browse/HIVE-21486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Ye updated HIVE-21486: Labels: lineage (was: ) > FinalSelectOps is empty in lineage index if there is a script > operator(transform) > - > > Key: HIVE-21486 > URL: https://issues.apache.org/jira/browse/HIVE-21486 > Project: Hive > Issue Type: Bug > Components: lineage >Affects Versions: 2.1.1, 2.3.4 >Reporter: Zihao Ye >Priority: Major > Labels: lineage > > SQL pattern: > create table t1 as select transform(c1) using '/bin/python script.py' as (c2) > from t2; > Lineage dependencies are correct. But the SelectOperator is not added to the > finalSelectOps in Lineage Index. So that index.getDependencies(finalSelOp) > got null in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21386) Extend the fetch task enhancement done in HIVE-21279 to make it work with query result cache
[
https://issues.apache.org/jira/browse/HIVE-21386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798110#comment-16798110
]
Hive QA commented on HIVE-21386:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 17 new + 757 unchanged - 0
fixed = 774 total (was 757) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
25s{color} | {color:red} ql generated 1 new + 2254 unchanged - 1 fixed = 2255
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to fakePathExists in
org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.setEntryValid(QueryResultsCache$CacheEntry,
FetchWork) At
QueryResultsCache.java:org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.setEntryValid(QueryResultsCache$CacheEntry,
FetchWork) At QueryResultsCache.java:[line 551] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16610/dev-support/hive-personality.sh
|
| git revision | master / 25b14be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16610/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16610/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16610/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Extend the fetch task enhancement done in HIVE-21279 to make it work with
> query result cache
>
>
> Key: HIVE-21386
> URL: https://issues.apache.org/jira/browse/HIVE-21386
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21386.1.patch, HIVE-21386.2.patch
>
>
> The improvement done in HIVE-21279 is disabled for query cache.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798105#comment-16798105
]
Naveen Gangam commented on HIVE-21484:
--
Should the HMS version be the version of the schema (from the HMS DB) or should
it be the version of the code they are running (like the fix you proposed) ?
Ideally, they should be the same but lets say given that HMS is standalone and
can have its own release schedule and that the schema version is not tied to
the release version anymore, then they could differ.
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798085#comment-16798085
]
Hive QA commented on HIVE-21446:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963206/HIVE-21446.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16609/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16609/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16609/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963206 - PreCommit-HIVE-Build
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch,
> HIVE-21446.03.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Attachment: HIVE-21467.02.patch > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch, HIVE-21467.02.patch, > HIVE-21467.02.patch > > > These imports trigger lots of warnings in ide, which could be annoying, and > it can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798065#comment-16798065 ] Laszlo Bodor commented on HIVE-21406: - green run > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798059#comment-16798059
]
Hive QA commented on HIVE-21446:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
19s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} shims/common: The patch generated 0 new + 94
unchanged - 1 fixed = 94 total (was 95) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch 0.23 passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 19 unchanged - 2 fixed
= 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16609/dev-support/hive-personality.sh
|
| git revision | master / 25b14be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16609/yetus/diff-checkstyle-ql.txt
|
| modules | C: shims/common shims/0.23 common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16609/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
>
[jira] [Comment Edited] (HIVE-15406) Consider vectorizing the new 'trunc' function
[
https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798002#comment-16798002
]
Laszlo Bodor edited comment on HIVE-15406 at 3/21/19 12:22 PM:
---
thanks for the review [~kgyrtkirk]
1. the zeros are the same as in nonvectorized execution, and it may be because
- somewhere while processing the input decimal - scale/precision is not enforced
{code}
selectExpressions: TruncDecimal(col 2, scale 0)(children:
CastDoubleToDecimal(col 0:double) -> 2:decimal(10,5)) -> 3:decimal(38,18)
{code}
2. attached the same queries in nonvectorized mode
i agree that it could be misleading to see leading zeros when the input is
casted to (10,5), but for now, it could remain as a sign that it should be
reviewed later (in both vec and nonvec implementation)
was (Author: abstractdog):
thanks for the review [~kgyrtkirk]
1. the zeros are the same as in nonvectorized execution, and it may be because
- somewhere while processing the input decimal - scale/precision is not enforced
2. attached the same queries in nonvectorized mode
i agree that it could be misleading to see leading zeros when the input is
casted to (10,5), but for now, it could remain as a sign that it should be
reviewed later (in both vec and nonvec implementation)
> Consider vectorizing the new 'trunc' function
> -
>
> Key: HIVE-15406
> URL: https://issues.apache.org/jira/browse/HIVE-15406
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 4.0.0
>Reporter: Matt McCline
>Assignee: Laszlo Bodor
>Priority: Critical
> Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch,
> HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch,
> HIVE-15406.06.patch
>
>
> Rounding function 'trunc' added by HIVE-14582.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-15406) Consider vectorizing the new 'trunc' function
[ https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798002#comment-16798002 ] Laszlo Bodor edited comment on HIVE-15406 at 3/21/19 12:21 PM: --- thanks for the review [~kgyrtkirk] 1. the zeros are the same as in nonvectorized execution, and it may be because - somewhere while processing the input decimal - scale/precision is not enforced 2. attached the same queries in nonvectorized mode i agree that it could be misleading to see leading zeros when the input is casted to (10,5), but for now, it could remain as a sign that it should be reviewed later (in both vec and nonvec implementation) was (Author: abstractdog): thanks for the review [~kgyrtkirk] 1. the zeros are the same as in nonvectorized execution, and it's because trunc function doesn't respect the input scale when it produces output, see: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L448 HiveDecimal has also a constructor with scale parameter, but it's not used 2. attached the same queries in nonvectorized mode i agree that it could be annoying to see leading zeros when the input is casted to (10,5), but for now, it could remain as a sign that it should be reviewed later (in both vec and nonvec implementation) > Consider vectorizing the new 'trunc' function > - > > Key: HIVE-15406 > URL: https://issues.apache.org/jira/browse/HIVE-15406 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 4.0.0 >Reporter: Matt McCline >Assignee: Laszlo Bodor >Priority: Critical > Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch, > HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch, > HIVE-15406.06.patch > > > Rounding function 'trunc' added by HIVE-14582. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798031#comment-16798031
]
Hive QA commented on HIVE-21484:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963198/HIVE-21484.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15840 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized
(batchId=261)
org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=261)
org.apache.hive.spark.client.TestSparkClient.testSyncRpc (batchId=331)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16608/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16608/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963198 - PreCommit-HIVE-Build
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15406) Consider vectorizing the new 'trunc' function
[ https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798002#comment-16798002 ] Laszlo Bodor commented on HIVE-15406: - thanks for the review [~kgyrtkirk] 1. the zeros are the same as in nonvectorized execution, and it's because trunc function doesn't respect the input scale when it produces output, see: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L448 HiveDecimal has also a constructor with scale parameter, but it's not used 2. attached the same queries in nonvectorized mode i agree that it could be annoying to see leading zeros when the input is casted to (10,5), but for now, it could remain as a sign that it should be reviewed later (in both vec and nonvec implementation) > Consider vectorizing the new 'trunc' function > - > > Key: HIVE-15406 > URL: https://issues.apache.org/jira/browse/HIVE-15406 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 4.0.0 >Reporter: Matt McCline >Assignee: Laszlo Bodor >Priority: Critical > Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch, > HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch, > HIVE-15406.06.patch > > > Rounding function 'trunc' added by HIVE-14582. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798004#comment-16798004
]
Hive QA commented on HIVE-21484:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 387 unchanged - 0 fixed = 388 total (was 387) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16608/dev-support/hive-personality.sh
|
| git revision | master / 25b14be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16608/yetus/diff-checkstyle-standalone-metastore_metastore-common.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16608/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: (was: OrcUtils.patch)
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HIVE-15406) Consider vectorizing the new 'trunc' function
[ https://issues.apache.org/jira/browse/HIVE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-15406: Attachment: HIVE-15406.06.patch > Consider vectorizing the new 'trunc' function > - > > Key: HIVE-15406 > URL: https://issues.apache.org/jira/browse/HIVE-15406 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 4.0.0 >Reporter: Matt McCline >Assignee: Laszlo Bodor >Priority: Critical > Attachments: HIVE-15406.01.patch, HIVE-15406.02.patch, > HIVE-15406.03.patch, HIVE-15406.04.patch, HIVE-15406.05.patch, > HIVE-15406.06.patch > > > Rounding function 'trunc' added by HIVE-14582. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21226) Exclude read-only transactions from ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-21226: - Assignee: Denys Kuzmenko > Exclude read-only transactions from ValidTxnList > > > Key: HIVE-21226 > URL: https://issues.apache.org/jira/browse/HIVE-21226 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Denys Kuzmenko >Priority: Major > > Once HIVE-21114 is done, we should make sure that ValidTxnList doesn't > contain any read-only txns in the exceptions list since by definition there > is no data tagged with such txnid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797986#comment-16797986
]
Denys Kuzmenko commented on HIVE-21397:
---
Attached patch
[HIVE-21397.patch|https://issues.apache.org/jira/secure/attachment/12963255/HIVE-21397.patch]
Should be submitted once
[ORC-477|https://issues.apache.org/jira/browse/ORC-477] is merged into
branch-1.5 and orc-core v1.5.5 is released.
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: HIVE-21397.patch
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: HIVE-21397.patch
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch, OrcUtils.patch,
> orc_file_dump.out, orc_file_dump.q
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
[jira] [Commented] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables
[
https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797981#comment-16797981
]
Hive QA commented on HIVE-21482:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963186/HIVE-21482.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16607/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16607/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16607/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963186 - PreCommit-HIVE-Build
> Partition discovery table property is added to non-partitioned external tables
> --
>
> Key: HIVE-21482
> URL: https://issues.apache.org/jira/browse/HIVE-21482
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21482.1.patch, HIVE-21482.2.patch
>
>
> Automatic partition discovery is added to external tables by default. But it
> doesn't check if the external table is partitioned or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: (was: HIVE-21397.patch)
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: (was: orc_file_dump.out)
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-9995:
-
Attachment: HIVE-9995.03.patch
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21397) BloomFilter for hive Managed [ACID] table does not work as expected
[
https://issues.apache.org/jira/browse/HIVE-21397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-21397:
--
Attachment: (was: orc_file_dump.q)
> BloomFilter for hive Managed [ACID] table does not work as expected
> ---
>
> Key: HIVE-21397
> URL: https://issues.apache.org/jira/browse/HIVE-21397
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Transactions
>Affects Versions: 3.1.1
>Reporter: vaibhav
>Assignee: Denys Kuzmenko
>Priority: Blocker
> Attachments: HIVE-21397.1.patch, HIVE-21397.patch
>
>
> Steps to Reproduce this issue :
> -
> 1. Create a HIveManaged table as below :
> -
> {code:java}
> CREATE TABLE `bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest;
>
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02',
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1551206683') {code}
> -
> 2. Insert a few rows.
> -
> -
> 3. Check if bloom filter or active : [ It does not show bloom filters for
> hive managed tables ]
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> [length: 791]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/managed/hive/bloomTest/delta_001_001_/bucket_0
> {code}
> -
> On Another hand: For hive External tables it works :
> -
> {code:java}
> CREATE external TABLE `ext_bloomTest`(
> `msisdn` string,
> `imsi` varchar(20),
> `imei` bigint,
> `cell_id` bigint)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
> 'bucketing_version'='2',
> 'orc.bloom.filter.columns'='msisdn,cell_id,imsi',
> 'orc.bloom.filter.fpp'='0.02') {code}
> -
> {code:java}
> [hive@c1162-node2 root]$ hive --orcfiledump
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> | grep -i bloom
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> Processing data file
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
> [length: 755]
> Structure for
> hdfs://c1162-node2.squadron-labs.com:8020/warehouse/tablespace/external/hive/ext_bloomTest/00_0
>
> Stream: column 1 section BLOOM_FILTER_UTF8 start: 41 length 110
> Stream: column 2 section BLOOM_FILTER_UTF8 start: 178 length 114
> Stream: column 4 section BLOOM_FILTER_UTF8 start: 340 length 109 {code}
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797975#comment-16797975
]
Denys Kuzmenko commented on HIVE-9995:
--
[~vgumashta] could you please review. Thanks you!
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-9995) ACID compaction tries to compact a single file
[
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797975#comment-16797975
]
Denys Kuzmenko edited comment on HIVE-9995 at 3/21/19 10:30 AM:
[~vgumashta] could you please review. Thank you!
was (Author: dkuzmenko):
[~vgumashta] could you please review. Thanks you!
> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch,
> HIVE-9995.03.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor
> compaction work to do. The system still tries to compact a single delta file
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:03
> /user/hive/warehouse/t/base_016
> -rw-r--r-- 1 ekoifman staff602 2016-06-09 16:03
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r-- 1 ekoifman staff514 2016-06-09 16:06
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with
> {noformat}
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/base_017
> -rw-r--r-- 1 ekoifman staff588 2016-06-09 16:07
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r-- 1 ekoifman staff500 2016-06-09 16:11
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x - ekoifman staff 0 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r-- 1 ekoifman staff612 2016-06-09 16:07
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21484) Metastore API getVersion() should return real version
[
https://issues.apache.org/jira/browse/HIVE-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797971#comment-16797971
]
Peter Vary commented on HIVE-21484:
---
+1 pending tests
> Metastore API getVersion() should return real version
> -
>
> Key: HIVE-21484
> URL: https://issues.apache.org/jira/browse/HIVE-21484
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-21484.01.patch
>
>
> Currently I see the {{getVersion}} implementation in the metastore is
> returning a hard-coded "3.0". It would be good to return the real version of
> the metastore server using {{HiveversionInfo}} so that clients can take
> certain actions based on metastore server versions.
> Possible use-cases are:
> 1. Client A can make use of new features introduced in given Metastore
> version else stick to the base functionality.
> 2. This version number can be used to do a version handshake between client
> and server in the future to improve our cross-version compatibity story.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21482) Partition discovery table property is added to non-partitioned external tables
[
https://issues.apache.org/jira/browse/HIVE-21482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797949#comment-16797949
]
Hive QA commented on HIVE-21482:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 47 unchanged - 2 fixed = 49 total (was 49) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16607/dev-support/hive-personality.sh
|
| git revision | master / 25b14be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16607/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql hbase-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16607/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Partition discovery table property is added to non-partitioned external tables
> --
>
> Key: HIVE-21482
> URL: https://issues.apache.org/jira/browse/HIVE-21482
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21482.1.patch, HIVE-21482.2.patch
>
>
> Automatic partition discovery is added to external tables by default. But it
> doesn't check if the external table is partitioned or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21473) Bumping jackson version
[ https://issues.apache.org/jira/browse/HIVE-21473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-21473: -- Attachment: HIVE-21473.3.patch > Bumping jackson version > --- > > Key: HIVE-21473 > URL: https://issues.apache.org/jira/browse/HIVE-21473 > Project: Hive > Issue Type: Task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-21473.2.patch, HIVE-21473.3.patch, HIVE-21473.patch > > > Bump jackson version to 2.9.8 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21205) Tests for replace flag in insert event messages in Metastore notifications.
[
https://issues.apache.org/jira/browse/HIVE-21205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797934#comment-16797934
]
Hive QA commented on HIVE-21205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963183/HIVE-21205.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16606/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16606/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16606/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963183 - PreCommit-HIVE-Build
> Tests for replace flag in insert event messages in Metastore notifications.
> ---
>
> Key: HIVE-21205
> URL: https://issues.apache.org/jira/browse/HIVE-21205
> Project: Hive
> Issue Type: Test
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
> Attachments: HIVE-21205.1.patch, HIVE-21205.2.patch
>
>
> The replace flag is initially added in HIVE-16197. It would be good to have
> some tests in TestDbNotificationListener to validate if the flag is set as
> expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21463) Table "partition_keys" has been specified with a primary-key to include column "TBL_ID"
[
https://issues.apache.org/jira/browse/HIVE-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797922#comment-16797922
]
Andy edited comment on HIVE-21463 at 3/21/19 8:49 AM:
--
The same question
Hadoop 3.1.2 Hive3.0.0
{code:java}
Caused by: org.datanucleus.exceptions.NucleusUserException: Table
"partition_keys" has been specified with a primary-key to include column
"TBL_ID" but this column is not found in the table. Please check your
column specification.
at
org.datanucleus.store.rdbms.table.ElementContainerTable.applyUserPrimaryKeySpecification(ElementContainerTable.java:225)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.table.CollectionTable.initialize(CollectionTable.java:235)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3279)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2889)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getPropertiesForGenerator(RDBMSStoreManager.java:2088)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1271)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3760)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.StateManagerImpl.setIdentity(StateManagerImpl.java:2267)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:484)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:120)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:218)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2079)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1923)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1778)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:724)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1301)
~[hive-exec-3.0.0.jar:3.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_66]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_66]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_66]
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
~[hive-exec-3.0.0.jar:3.0.0]
at com.sun.proxy.$Proxy36.createTable(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1824)
~[hive-exec-3.0.0.jar:3.0.0]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1724)
~[hive-exec-3.0.0.jar:3.0.0]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1952)
~[hive-exec-3.0.0.jar:3.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_66]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_66]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_66]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-3.0.0.jar:3.0.0]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-3.0.0.jar:3.0.0]
at com.sun.proxy.$Proxy38.create_table_with_environment_context(Unknown Source)
~[?:?]
at
[jira] [Commented] (HIVE-21463) Table "partition_keys" has been specified with a primary-key to include column "TBL_ID"
[
https://issues.apache.org/jira/browse/HIVE-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797922#comment-16797922
]
Andy commented on HIVE-21463:
-
The same question
Hadoop 3.1.2 Hive3.1.1
{code:java}
Caused by: org.datanucleus.exceptions.NucleusUserException: Table
"partition_keys" has been specified with a primary-key to include column
"TBL_ID" but this column is not found in the table. Please check your
column specification.
at
org.datanucleus.store.rdbms.table.ElementContainerTable.applyUserPrimaryKeySpecification(ElementContainerTable.java:225)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.table.CollectionTable.initialize(CollectionTable.java:235)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3279)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2889)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1627)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:672)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.rdbms.RDBMSStoreManager.getPropertiesForGenerator(RDBMSStoreManager.java:2088)
~[datanucleus-rdbms-4.1.19.jar:?]
at
org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1271)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3760)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.StateManagerImpl.setIdentity(StateManagerImpl.java:2267)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:484)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.StateManagerImpl.initialiseForPersistentNew(StateManagerImpl.java:120)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:218)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2079)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1923)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1778)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
~[datanucleus-core-4.1.17.jar:?]
at
org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:724)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:1301)
~[hive-exec-3.0.0.jar:3.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_66]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_66]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_66]
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
~[hive-exec-3.0.0.jar:3.0.0]
at com.sun.proxy.$Proxy36.createTable(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1824)
~[hive-exec-3.0.0.jar:3.0.0]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1724)
~[hive-exec-3.0.0.jar:3.0.0]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1952)
~[hive-exec-3.0.0.jar:3.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_66]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_66]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_66]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-3.0.0.jar:3.0.0]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-3.0.0.jar:3.0.0]
at com.sun.proxy.$Proxy38.create_table_with_environment_context(Unknown Source)
~[?:?]
at
