[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[
https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430846#comment-16430846
]
Hive QA commented on HIVE-18839:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
57s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} ql: The patch generated 41 new + 1161 unchanged - 20
fixed = 1202 total (was 1181) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} standalone-metastore: The patch generated 10 new +
1603 unchanged - 1 fixed = 1613 total (was 1604) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10094/dev-support/hive-personality.sh
|
| git revision | master / 959e772 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/patch-asflicense-problems.txt
|
| modules | C: common itests ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement incremental rebuild for materialized views (only insert operations
> in source tables)
> --
>
> Key: HIVE-18839
> URL: https://issues.apache.org/jira/browse/HIVE-18839
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: TODOC3.0
> Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch,
> HIVE-18839.03.patch,
[jira] [Updated] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[ https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19014: -- Attachment: HIVE-19014.07.patch > utilize YARN-8028 (queue ACL check) in Hive Tez session pool > > > Key: HIVE-19014 > URL: https://issues.apache.org/jira/browse/HIVE-19014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19014.01.patch, HIVE-19014.02.patch, > HIVE-19014.03.patch, HIVE-19014.04.patch, HIVE-19014.05.patch, > HIVE-19014.06.patch, HIVE-19014.07.patch, HIVE-19014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[ https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430943#comment-16430943 ] Jason Dere commented on HIVE-19014: --- re-attaching same patch as HIVE-19014.07.patch > utilize YARN-8028 (queue ACL check) in Hive Tez session pool > > > Key: HIVE-19014 > URL: https://issues.apache.org/jira/browse/HIVE-19014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19014.01.patch, HIVE-19014.02.patch, > HIVE-19014.03.patch, HIVE-19014.04.patch, HIVE-19014.05.patch, > HIVE-19014.06.patch, HIVE-19014.07.patch, HIVE-19014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430713#comment-16430713
]
Zoltan Haindrich commented on HIVE-19097:
-
[~jcamachorodriguez] I've taken a look at HivePointLookupOptimizerRule.java ; I
think it would be better to place my IN opener there...because that's also a
calcite rule; I don't really need to do any modifications ; however the change
still doesn't worth that much - even thru the in is now gone; I think there is
no rule which could simplify ((a=1 || b=2) && a=1) to (a=1)...or I missed it :)
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19129) Support DEFAULT keyword with MERGE
[ https://issues.apache.org/jira/browse/HIVE-19129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430865#comment-16430865 ] Ashutosh Chauhan commented on HIVE-19129: - +1 > Support DEFAULT keyword with MERGE > -- > > Key: HIVE-19129 > URL: https://issues.apache.org/jira/browse/HIVE-19129 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19129.1.patch > > > HIVE-19059 added support for DEFAULT keyword in INSERT INTO (VALUES) and > UPDATE. MERGE is pending. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[ https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19133: Component/s: Web UI HiveServer2 > HS2 WebUI phase-wise performance metrics not showing correctly > -- > > Key: HIVE-19133 > URL: https://issues.apache.org/jira/browse/HIVE-19133 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Web UI >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: WebUI-compile time query metrics.png > > > The query specific WebUI metrics (go to drilldown -> performance logging) are > not showing up in the correct phase and are often mixed up. > Attaching screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19127) Concurrency fixes in QueryResultsCache
[ https://issues.apache.org/jira/browse/HIVE-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19127: -- Attachment: HIVE-19127.2.patch > Concurrency fixes in QueryResultsCache > -- > > Key: HIVE-19127 > URL: https://issues.apache.org/jira/browse/HIVE-19127 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19127.1.patch, HIVE-19127.2.patch > > > A few fixes around concurrent access in the results cache > - Take a lock on the cache entry when in the process of setting the cache > entry from PENDING state to VALID state, so that other threads cannot > invalidate the entry > - The write lock on the cache was not being taken when removing an entry from > the cache. > - synchronize access when iterating through the lru keyset -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[ https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19133: Attachment: HIVE-19133.1.patch > HS2 WebUI phase-wise performance metrics not showing correctly > -- > > Key: HIVE-19133 > URL: https://issues.apache.org/jira/browse/HIVE-19133 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Web UI >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-19133.1.patch, WebUI-compile time query metrics.png > > > The query specific WebUI metrics (go to drilldown -> performance logging) are > not showing up in the correct phase and are often mixed up. > Attaching screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[ https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19133 started by Bharathkrishna Guruvayoor Murali. --- > HS2 WebUI phase-wise performance metrics not showing correctly > -- > > Key: HIVE-19133 > URL: https://issues.apache.org/jira/browse/HIVE-19133 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Web UI >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: WebUI-compile time query metrics.png > > > The query specific WebUI metrics (go to drilldown -> performance logging) are > not showing up in the correct phase and are often mixed up. > Attaching screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[ https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-19133: --- > HS2 WebUI phase-wise performance metrics not showing correctly > -- > > Key: HIVE-19133 > URL: https://issues.apache.org/jira/browse/HIVE-19133 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: WebUI-compile time query metrics.png > > > The query specific WebUI metrics (go to drilldown -> performance logging) are > not showing up in the correct phase and are often mixed up. > Attaching screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19127) Concurrency fixes in QueryResultsCache
[ https://issues.apache.org/jira/browse/HIVE-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430949#comment-16430949 ] Jason Dere commented on HIVE-19127: --- re-attaching same patch to kick off ptests. > Concurrency fixes in QueryResultsCache > -- > > Key: HIVE-19127 > URL: https://issues.apache.org/jira/browse/HIVE-19127 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19127.1.patch, HIVE-19127.2.patch > > > A few fixes around concurrent access in the results cache > - Take a lock on the cache entry when in the process of setting the cache > entry from PENDING state to VALID state, so that other threads cannot > invalidate the entry > - The write lock on the cache was not being taken when removing an entry from > the cache. > - synchronize access when iterating through the lru keyset -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[
https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430954#comment-16430954
]
Hive QA commented on HIVE-18839:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918121/HIVE-18839.03.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 149 failed/errored test(s), 13566 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestExportImport - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-18859) Incorrect handling of thrift metastore exceptions
[ https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18859: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Ganesha! > Incorrect handling of thrift metastore exceptions > - > > Key: HIVE-18859 > URL: https://issues.apache.org/jira/browse/HIVE-18859 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.1.1 >Reporter: Ganesha Shreedhara >Assignee: Ganesha Shreedhara >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18859.patch > > > Currently any run time exception thrown in thrift metastore during the > following operations is not getting sent to hive execution engine. > * grant/revoke role > * grant/revoke privileges > * create role > This is because ThriftHiveMetastore just handles MetaException and throws > TException during the processing of these requests. So, the command just > fails at thrift metastore end when there is run time exception (Exception can > be seen in metastore log) but the hive execution engine will keep on waiting > for the response from thrift metatstore. > > Steps to reproduce this problem : > Launch thrift metastore > Launch hive cli by passing --hiveconf > hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host > and port) > Execute the following commands: > # set role admin > # create role test; (succeeds) > # create role test; ( hive version 2.1.1 : command is stuck, waiting for the > response from thrift metastore; hive version 1.2.1: command fails with > exception as null) > > I have uploaded the patch which has the fix in which I am handling the > checked exceptions in MetaException and throwing unchecked exceptions using > TException which fixes the problem. Please review and suggest if there is a > better way of handling this issue. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-10491) Refactor HBaseStorageHandler::configureJobConf() and configureTableJobProperties
[
https://issues.apache.org/jira/browse/HIVE-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430782#comment-16430782
]
BELUGA BEHR commented on HIVE-10491:
{code}
/*
* HIVE-6356
* The following code change is only needed for hbase-0.96.0 due to
HBASE-9165, and
* will not be required once Hive bumps up its hbase version). At that
time , we will
* only need TableMapReduceUtil.addDependencyJars(jobConf) here.
*/
{code}
https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java#L338-L343
> Refactor HBaseStorageHandler::configureJobConf() and
> configureTableJobProperties
>
>
> Key: HIVE-10491
> URL: https://issues.apache.org/jira/browse/HIVE-10491
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Ashutosh Chauhan
>Assignee: Swarnim Kulkarni
>Priority: Major
>
> 3 tasks as a part of this refactor:
> * Bump hbase version to 1.x
> * Remove HIVE-6356 hack for counter class from configureJobConf()
> * Make use of TableMapReduceUtil.initTableSnapshotMapperJob() instead of
> manually doing steps done in that method in configureTableJobProperties()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[
https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430678#comment-16430678
]
Ashutosh Chauhan commented on HIVE-18839:
-
+1 pending tests
> Implement incremental rebuild for materialized views (only insert operations
> in source tables)
> --
>
> Key: HIVE-18839
> URL: https://issues.apache.org/jira/browse/HIVE-18839
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: TODOC3.0
> Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch,
> HIVE-18839.03.patch, HIVE-18839.patch
>
>
> Implementation will follow current code path for full rebuild.
> When the MV query plan is retrieved, if the MV contents are outdated because
> there were insert operations in the source tables, we will introduce a filter
> with a condition based on stored value of ValidWriteIdLists. For instance,
> {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the
> rewriting will do the rest of the work by creating a partial rewriting, where
> the contents of the MV are read as well as the new contents from the source
> tables.
> This mechanism will not work only for ALTER MV... REBUILD, but also for user
> queries which will be able to benefit from using outdated MVs to compute part
> of the needed results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19097) related equals and in operators may cause inaccurate stats estimations
[
https://issues.apache.org/jira/browse/HIVE-19097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430724#comment-16430724
]
Jesus Camacho Rodriguez commented on HIVE-19097:
I thought the different versions of {{ReduceExpressions}} rules would do the
trick (relying on {{RexSimplify}}). There were similar simplifications for such
cases, including {{AND}}, {{OR}}, simplification of ranges, etc., but maybe we
are missing the case you are mentioning for some reason. If we do, probably
{{RexSimplify}} is the class to extend to accomplish this goal.
> related equals and in operators may cause inaccurate stats estimations
> --
>
> Key: HIVE-19097
> URL: https://issues.apache.org/jira/browse/HIVE-19097
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19097.01.patch
>
>
> tpcds#74 is optimized in a way that for date_dim the condition contains IN
> and = for the same column
> {code:java}
> | Map Operator Tree: |
> | TableScan |
> | alias: date_dim |
> | filterExpr: (((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) or ((d_year) IN (2001, 2002) and (d_year =
> 2001) and d_date_sk is not null)) (type: boolean) |
> | Statistics: Num rows: 73049 Data size: 876588 Basic
> stats: COMPLETE Column stats: COMPLETE |
> | Filter Operator |
> | predicate: ((d_year) IN (2001, 2002) and (d_year =
> 2002) and d_date_sk is not null) (type: boolean) |
> | Statistics: Num rows: 4 Data size: 48 Basic stats:
> COMPLETE Column stats: COMPLETE |
> {code}
> the "real" row count will be 365
> for separate {{IN}} and {{=}} the estimation is very good; but if both are
> present it becomes (very) underestimated.
> {code:java}
> set hive.query.results.cache.enabled=false;
> drop table if exists t1;
> drop table if exists t8;
> create table t1 (a integer,b integer);
> create table t8 like t1;
> insert into t1 values (1,1),(2,2),(3,3),(4,4),(5,5);
> insert into t8
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1 union all
> select * from t1 union all select * from t1 union all select * from t1 union
> all select * from t1
> ;
> analyze table t1 compute statistics for columns;
> analyze table t8 compute statistics for columns;
> explain analyze select sum(a) from t8 where b in (2,3) group by b;
> explain analyze select sum(a) from t8 where b=2 group by b;
> explain analyze select sum(a) from t1 where b in (2,3) and b=2 group by b;
> explain analyze select sum(a) from t8 where b in (2,3) and b=2 group by b;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18783) ALTER TABLE post-commit listener does not include the transactional listener responses
[
https://issues.apache.org/jira/browse/HIVE-18783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-18783:
---
Fix Version/s: 2.4.0
> ALTER TABLE post-commit listener does not include the transactional listener
> responses
> ---
>
> Key: HIVE-18783
> URL: https://issues.apache.org/jira/browse/HIVE-18783
> Project: Hive
> Issue Type: Bug
>Reporter: Na Li
>Assignee: Sergio Peña
>Priority: Major
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-18783.1.patch, HIVE-18783.2.patch,
> HIVE-18783.3.patch
>
>
> in HiveMetaStore, alter_table_core does NOT call transactional listener, and
> the notification ID corresponding to the alter table event is NOT set in the
> event parameters.
> {code}
> + alter_table_core
>
> try {
> Table oldt = this.get_table_core(dbname, name);
> this.firePreEvent(new PreAlterTableEvent(oldt, newTable, this));
> this.alterHandler.alterTable(this.getMS(), this.wh, dbname, name,
> newTable, envContext, this);
> success = true;
> if (!this.listeners.isEmpty()) {
> MetaStoreListenerNotifier.notifyEvent(this.listeners,
> EventType.ALTER_TABLE, new AlterTableEvent(oldt, newTable, true, this),
> envContext);
> }
> } catch (NoSuchObjectException var12) {
> ex = var12;
> throw new InvalidOperationException(var12.getMessage());
> } catch (Exception var13) {
> ex = var13;
> if (var13 instanceof MetaException) {
> throw (MetaException)var13;
> }
> if (var13 instanceof InvalidOperationException) {
> throw (InvalidOperationException)var13;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18991) Drop database cascade doesn't work with materialized views
[ https://issues.apache.org/jira/browse/HIVE-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430663#comment-16430663 ] Ashutosh Chauhan commented on HIVE-18991: - Some of failed junit metastore tests look related. > Drop database cascade doesn't work with materialized views > -- > > Key: HIVE-18991 > URL: https://issues.apache.org/jira/browse/HIVE-18991 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18991.01.patch, HIVE-18991.03.patch, > HIVE-18991.06.patch, HIVE-18991.07.patch, HIVE-18991.patch > > > Create a database, add a table and then a materialized view that depends on > the table. Then drop the database with cascade set. Sometimes this will > fail because when HiveMetaStore.drop_database_core goes to drop all of the > tables it may drop the base table before the materialized view, which will > cause an integrity constraint violation in the RDBMS. To resolve this that > method should change to fetch and drop materialized views before tables. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19129) Support DEFAULT keyword with MERGE
[
https://issues.apache.org/jira/browse/HIVE-19129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430125#comment-16430125
]
Hive QA commented on HIVE-19129:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 2 new + 612 unchanged - 0
fixed = 614 total (was 612) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10088/dev-support/hive-personality.sh
|
| git revision | master / a263f08 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10088/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10088/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10088/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support DEFAULT keyword with MERGE
> --
>
> Key: HIVE-19129
> URL: https://issues.apache.org/jira/browse/HIVE-19129
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19129.1.patch
>
>
> HIVE-19059 added support for DEFAULT keyword in INSERT INTO (VALUES) and
> UPDATE. MERGE is pending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430974#comment-16430974
]
Bharathkrishna Guruvayoor Murali commented on HIVE-18928:
-
I was working on another Jira related to PerfLogger behaving differently with
multiple threads (HIVE-19133)
I could see that PerfLogger is passed between background and handler threads
here
[https://github.com/apache/hive/blob/a2394c5bf2e5b257a3a1b3e2bda4a25b92005315/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L294]
Do you think that is the reason for this race condition as well?
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19134) Update copyright NOTICE and fix rat check failures
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19134: --- Attachment: HIVE-19134.2.patch > Update copyright NOTICE and fix rat check failures > -- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch, HIVE-19134.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19074) Vectorization: Add llap vectorization_div0.q.out Q output file
[ https://issues.apache.org/jira/browse/HIVE-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19074: Attachment: (was: HIVE-19074.04.patch) > Vectorization: Add llap vectorization_div0.q.out Q output file > -- > > Key: HIVE-19074 > URL: https://issues.apache.org/jira/browse/HIVE-19074 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19074.02.patch, HIVE-19074.03.patch, > HIVE-19074.05.patch > > > At some point llap/vectorization_div0.q.out got omitted. > The Q file output is unstable because of missing ORDER BY columns. You must > have ORDER BY on all/critical the columns when there is a LIMIT clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19074) Vectorization: Add llap vectorization_div0.q.out Q output file
[ https://issues.apache.org/jira/browse/HIVE-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19074: Attachment: HIVE-19074.05.patch > Vectorization: Add llap vectorization_div0.q.out Q output file > -- > > Key: HIVE-19074 > URL: https://issues.apache.org/jira/browse/HIVE-19074 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19074.02.patch, HIVE-19074.03.patch, > HIVE-19074.05.patch > > > At some point llap/vectorization_div0.q.out got omitted. > The Q file output is unstable because of missing ORDER BY columns. You must > have ORDER BY on all/critical the columns when there is a LIMIT clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19074) Vectorization: Add llap vectorization_div0.q.out Q output file
[ https://issues.apache.org/jira/browse/HIVE-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19074: Attachment: (was: HIVE-19074.05.patch) > Vectorization: Add llap vectorization_div0.q.out Q output file > -- > > Key: HIVE-19074 > URL: https://issues.apache.org/jira/browse/HIVE-19074 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19074.02.patch, HIVE-19074.03.patch, > HIVE-19074.05.patch > > > At some point llap/vectorization_div0.q.out got omitted. > The Q file output is unstable because of missing ORDER BY columns. You must > have ORDER BY on all/critical the columns when there is a LIMIT clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19074) Vectorization: Add llap vectorization_div0.q.out Q output file
[ https://issues.apache.org/jira/browse/HIVE-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19074: Attachment: HIVE-19074.05.patch > Vectorization: Add llap vectorization_div0.q.out Q output file > -- > > Key: HIVE-19074 > URL: https://issues.apache.org/jira/browse/HIVE-19074 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19074.02.patch, HIVE-19074.03.patch, > HIVE-19074.05.patch > > > At some point llap/vectorization_div0.q.out got omitted. > The Q file output is unstable because of missing ORDER BY columns. You must > have ORDER BY on all/critical the columns when there is a LIMIT clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-19135: - > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431094#comment-16431094
]
Hive QA commented on HIVE-18946:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918161/HIVE-18946.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 100 failed/errored test(s), 13188 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=252)
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=246)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
TestExportImport - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=246)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=252)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=252)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-19127) Concurrency fixes in QueryResultsCache
[ https://issues.apache.org/jira/browse/HIVE-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19127: -- Status: Patch Available (was: Open) > Concurrency fixes in QueryResultsCache > -- > > Key: HIVE-19127 > URL: https://issues.apache.org/jira/browse/HIVE-19127 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19127.1.patch, HIVE-19127.2.patch > > > A few fixes around concurrent access in the results cache > - Take a lock on the cache entry when in the process of setting the cache > entry from PENDING state to VALID state, so that other threads cannot > invalidate the entry > - The write lock on the cache was not being taken when removing an entry from > the cache. > - synchronize access when iterating through the lru keyset -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16944) schematool -dbType hive should give some more feedback/assistance
[
https://issues.apache.org/jira/browse/HIVE-16944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431012#comment-16431012
]
Bharathkrishna Guruvayoor Murali commented on HIVE-16944:
-
[~pvary] Thanks for pointing that out. In that case, I think the change is not
appropriate here, we cannot exit because there could be values provided for url
and driver through the config as well.
So I guess a code change is needed only if it is throwing a NPE which I could
not reproduce.
[~cartershanklin] can you make sure if this NPE is still happening?
> schematool -dbType hive should give some more feedback/assistance
> -
>
> Key: HIVE-16944
> URL: https://issues.apache.org/jira/browse/HIVE-16944
> Project: Hive
> Issue Type: Bug
>Reporter: Carter Shanklin
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-16944.1.patch
>
>
> Given the other ways schematool is used, the most obvious guess I would have
> for initializing the Hive schema is:
> {code}
> schematool -metaDbType mysql -dbType hive -initSchema
> {code}
> Unfortunately that fails with this NPE:
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getDbCommandParser(HiveSchemaHelper.java:570)
> at
> org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getDbCommandParser(HiveSchemaHelper.java:564)
> at
> org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getDbCommandParser(HiveSchemaHelper.java:560)
> at
> org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper$HiveCommandParser.(HiveSchemaHelper.java:373)
> at
> org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getDbCommandParser(HiveSchemaHelper.java:573)
> at
> org.apache.hive.beeline.HiveSchemaTool.getDbCommandParser(HiveSchemaTool.java:165)
> at
> org.apache.hive.beeline.HiveSchemaTool.(HiveSchemaTool.java:101)
> at org.apache.hive.beeline.HiveSchemaTool.(HiveSchemaTool.java:90)
> at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
> Two additional arguments are needed:
> -url jdbc:hive2://localhost:1/default -driver
> org.apache.hive.jdbc.HiveDriver
> If the user does not supply these for dbType hive, schematool should detect
> and error out appropriately, plus give an example of what it's looking for.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19105) HIVE-18781 broke WarehouseInstance
[
https://issues.apache.org/jira/browse/HIVE-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-19105:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed. Thank you Thejas for the review.
> HIVE-18781 broke WarehouseInstance
> --
>
> Key: HIVE-19105
> URL: https://issues.apache.org/jira/browse/HIVE-19105
> Project: Hive
> Issue Type: Bug
> Components: repl, Tests
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19105.patch
>
>
> HIVE-18781 removed the line
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> from WarehouseInstance. This causes tests such as
> TestReplicationOnHDFSEncryptedZones and TestExportImport to hang as they are
> trying to connect to a non-existent ZooKeeper instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19105) HIVE-18781 broke WarehouseInstance
[
https://issues.apache.org/jira/browse/HIVE-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430961#comment-16430961
]
Thejas M Nair commented on HIVE-19105:
--
+1
I see that these tests are no longer hanging after this change.
> HIVE-18781 broke WarehouseInstance
> --
>
> Key: HIVE-19105
> URL: https://issues.apache.org/jira/browse/HIVE-19105
> Project: Hive
> Issue Type: Bug
> Components: repl, Tests
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19105.patch
>
>
> HIVE-18781 removed the line
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> from WarehouseInstance. This causes tests such as
> TestReplicationOnHDFSEncryptedZones and TestExportImport to hang as they are
> trying to connect to a non-existent ZooKeeper instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-14388: Status: In Progress (was: Patch Available) > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18857) Store default value text instead of default value expression in metastore
[ https://issues.apache.org/jira/browse/HIVE-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18857: --- Status: Open (was: Patch Available) > Store default value text instead of default value expression in metastore > - > > Key: HIVE-18857 > URL: https://issues.apache.org/jira/browse/HIVE-18857 > Project: Hive > Issue Type: Improvement > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18857.1.patch, HIVE-18857.2.patch, > HIVE-18857.3.patch > > > Currently for default value an expression is generated and serialized to > store in metastore. It should be improved to rather serialize the default > value itself instead of expression and store that in metastore. This will > have the following benefits: > * It will make metastore schema upgrade safe. e.g. if a UDF function name is > changed hive wouldn't be able to parse back the expression for this UDF which > was serialized in earlier version. > * It will make metastore schema for default constraint hive agnostic. Other > databases would be able to use the value as it is. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18857) Store default value text instead of default value expression in metastore
[ https://issues.apache.org/jira/browse/HIVE-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18857: --- Status: Patch Available (was: Open) > Store default value text instead of default value expression in metastore > - > > Key: HIVE-18857 > URL: https://issues.apache.org/jira/browse/HIVE-18857 > Project: Hive > Issue Type: Improvement > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18857.1.patch, HIVE-18857.2.patch, > HIVE-18857.3.patch > > > Currently for default value an expression is generated and serialized to > store in metastore. It should be improved to rather serialize the default > value itself instead of expression and store that in metastore. This will > have the following benefits: > * It will make metastore schema upgrade safe. e.g. if a UDF function name is > changed hive wouldn't be able to parse back the expression for this UDF which > was serialized in earlier version. > * It will make metastore schema for default constraint hive agnostic. Other > databases would be able to use the value as it is. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19134) Update copyright NOTICE
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430994#comment-16430994 ] Vineet Garg commented on HIVE-19134: [~ashutoshc] can you take a look? > Update copyright NOTICE > --- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19134) Update copyright NOTICE
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-19134: -- > Update copyright NOTICE > --- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19134) Update copyright NOTICE
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19134: --- Attachment: HIVE-19134.1.patch > Update copyright NOTICE > --- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19134) Update copyright NOTICE
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19134: --- Status: Patch Available (was: Open) > Update copyright NOTICE > --- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-17645:
--
Attachment: HIVE-17645.1.patch
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19127) Concurrency fixes in QueryResultsCache
[ https://issues.apache.org/jira/browse/HIVE-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19127: -- Status: Open (was: Patch Available) > Concurrency fixes in QueryResultsCache > -- > > Key: HIVE-19127 > URL: https://issues.apache.org/jira/browse/HIVE-19127 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19127.1.patch, HIVE-19127.2.patch > > > A few fixes around concurrent access in the results cache > - Take a lock on the cache entry when in the process of setting the cache > entry from PENDING state to VALID state, so that other threads cannot > invalidate the entry > - The write lock on the cache was not being taken when removing an entry from > the cache. > - synchronize access when iterating through the lru keyset -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19129) Support DEFAULT keyword with MERGE
[ https://issues.apache.org/jira/browse/HIVE-19129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19129: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master > Support DEFAULT keyword with MERGE > -- > > Key: HIVE-19129 > URL: https://issues.apache.org/jira/browse/HIVE-19129 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19129.1.patch > > > HIVE-19059 added support for DEFAULT keyword in INSERT INTO (VALUES) and > UPDATE. MERGE is pending. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[
https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430996#comment-16430996
]
Ashutosh Chauhan commented on HIVE-18839:
-
Failures TestMetaStoreMaterializationsCacheCleaner and
TestUpdateDeleteSemanticAnalyzer looks related.
> Implement incremental rebuild for materialized views (only insert operations
> in source tables)
> --
>
> Key: HIVE-18839
> URL: https://issues.apache.org/jira/browse/HIVE-18839
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: TODOC3.0
> Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch,
> HIVE-18839.03.patch, HIVE-18839.patch
>
>
> Implementation will follow current code path for full rebuild.
> When the MV query plan is retrieved, if the MV contents are outdated because
> there were insert operations in the source tables, we will introduce a filter
> with a condition based on stored value of ValidWriteIdLists. For instance,
> {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the
> rewriting will do the rest of the work by creating a partial rewriting, where
> the contents of the MV are read as well as the new contents from the source
> tables.
> This mechanism will not work only for ALTER MV... REBUILD, but also for user
> queries which will be able to benefit from using outdated MVs to compute part
> of the needed results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430995#comment-16430995
]
Gopal V commented on HIVE-18928:
Yes, that is the reason this is broken - the handler is reused before the
background thread exits.
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19134) Update copyright NOTICE and fix rat check failures
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431006#comment-16431006 ] Vineet Garg commented on HIVE-19134: [~ashutoshc] Can you take a look at second patch? This one contains a minor fix for missing apache license header. > Update copyright NOTICE and fix rat check failures > -- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch, HIVE-19134.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431042#comment-16431042
]
Jason Dere commented on HIVE-17645:
---
Adding a patch to try to replace the usage of SessionState.getTxnMgr() in the
non-test code. There are still a couple of tests that use
SessionState.getTxnMgr(), so I have not removed this yet.
[~ekoifman] [~sershe] does this look ok?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19105) HIVE-18781 broke WarehouseInstance
[
https://issues.apache.org/jira/browse/HIVE-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430960#comment-16430960
]
Thejas M Nair commented on HIVE-19105:
--
[~sankarh]
Can you please review ?
cc [~maheshk114]
I see these tests were consistently failing to run in HIVE-18781.
> HIVE-18781 broke WarehouseInstance
> --
>
> Key: HIVE-19105
> URL: https://issues.apache.org/jira/browse/HIVE-19105
> Project: Hive
> Issue Type: Bug
> Components: repl, Tests
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19105.patch
>
>
> HIVE-18781 removed the line
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> from WarehouseInstance. This causes tests such as
> TestReplicationOnHDFSEncryptedZones and TestExportImport to hang as they are
> trying to connect to a non-existent ZooKeeper instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19134) Update copyright NOTICE
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431000#comment-16431000 ] Ashutosh Chauhan commented on HIVE-19134: - +1 no need to run tests. > Update copyright NOTICE > --- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19134.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431003#comment-16431003
]
Bharathkrishna Guruvayoor Murali commented on HIVE-18928:
-
[~gopalv] I have seen this same reason causing HIVE-19133 as well.
But I am not sure why the parent perfLogger is being passed, I can't see any
specific use it has.
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19119) Fix the TestAppendPartitions tests which are failing in the pre-commit runs
[ https://issues.apache.org/jira/browse/HIVE-19119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19119: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Marta for the patch. > Fix the TestAppendPartitions tests which are failing in the pre-commit runs > --- > > Key: HIVE-19119 > URL: https://issues.apache.org/jira/browse/HIVE-19119 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-19119.1.patch > > > The test got fixed in > [HIVE-19060|https://issues.apache.org/jira/browse/HIVE-19060], but the fix > got overwritten by an other commit, so the testAppendPartitionNullPartValues > and testAppendPartitionEmptyPartValues test cases are failing again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[
https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431136#comment-16431136
]
Hive QA commented on HIVE-19014:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 12 new + 428 unchanged - 0
fixed = 440 total (was 428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10096/dev-support/hive-personality.sh
|
| git revision | master / d589ee7 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10096/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10096/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10096/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> utilize YARN-8028 (queue ACL check) in Hive Tez session pool
>
>
> Key: HIVE-19014
> URL: https://issues.apache.org/jira/browse/HIVE-19014
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19014.01.patch, HIVE-19014.02.patch,
> HIVE-19014.03.patch, HIVE-19014.04.patch, HIVE-19014.05.patch,
> HIVE-19014.06.patch, HIVE-19014.07.patch, HIVE-19014.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19136) DbNotifications clean up throws NPE on mysql databases
[
https://issues.apache.org/jira/browse/HIVE-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-19136:
--
> DbNotifications clean up throws NPE on mysql databases
> --
>
> Key: HIVE-19136
> URL: https://issues.apache.org/jira/browse/HIVE-19136
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> I see following stack trace in HMS logs when db notification cleaner thread
> tries to clean the old notification.
> {noformat}
> Exception in thread "CleanerThread" javax.jdo.JDODataStoreException:
> Transaction failed to commit
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at
> org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:189)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:790)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:10425)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:9258)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy22.cleanNotificationEvents(Unknown Source)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener$CleanerThread.run(DbNotificationListener.java:737)
> NestedThrowablesStackTrace:
> Unexpected exception encountered during query.
> org.datanucleus.exceptions.NucleusDataStoreException: Unexpected exception
> encountered during query.
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:569)
> at
> org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionRolledBack(ConnectionManagerImpl.java:400)
> at
> org.datanucleus.TransactionImpl.internalRollback(TransactionImpl.java:534)
> at org.datanucleus.TransactionImpl.rollback(TransactionImpl.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.rollback(JDOTransaction.java:185)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.rollbackTransaction(ObjectStore.java:790)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.rollbackAndCleanup(ObjectStore.java:10425)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:9258)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy22.cleanNotificationEvents(Unknown Source)
> at
> org.apache.hive.hcatalog.listener.DbNotificationListener$CleanerThread.run(DbNotificationListener.java:737)
> Caused by: java.sql.SQLException: Unexpected exception encountered during
> query.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2582)
> at
> com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4698)
> at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4590)
> at
> com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:233)
> at
> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:557)
> ... 14 more
> Caused by: java.lang.NullPointerException
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
> ... 18 more
> Nested Throwables StackTrace:
> java.sql.SQLException: Unexpected exception encountered during query.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:963)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
> at
[jira] [Updated] (HIVE-19130) NPE is thrown when REPL LOAD applied drop partition event.
[ https://issues.apache.org/jira/browse/HIVE-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19130: --- Fix Version/s: (was: 3.0.0) 3.1.0 > NPE is thrown when REPL LOAD applied drop partition event. > -- > > Key: HIVE-19130 > URL: https://issues.apache.org/jira/browse/HIVE-19130 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication > Fix For: 3.1.0 > > > During incremental replication, if we split the events batch as follows, then > the REPL LOAD on second batch throws NPE. > Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1) > Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> > DROP_PARTITION (t1.p1) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19134) Update copyright NOTICE and fix rat check failures
[ https://issues.apache.org/jira/browse/HIVE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19134: --- Fix Version/s: 3.1.0 > Update copyright NOTICE and fix rat check failures > -- > > Key: HIVE-19134 > URL: https://issues.apache.org/jira/browse/HIVE-19134 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19134.1.patch, HIVE-19134.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19135: -- Priority: Blocker (was: Major) > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.0.0 > > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19130) NPE is thrown when REPL LOAD applied drop partition event.
[ https://issues.apache.org/jira/browse/HIVE-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431200#comment-16431200 ] Vineet Garg commented on HIVE-19130: Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > NPE is thrown when REPL LOAD applied drop partition event. > -- > > Key: HIVE-19130 > URL: https://issues.apache.org/jira/browse/HIVE-19130 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, Replication > Fix For: 3.1.0 > > > During incremental replication, if we split the events batch as follows, then > the REPL LOAD on second batch throws NPE. > Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1) > Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> > DROP_PARTITION (t1.p1) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19135: -- Fix Version/s: 3.0.0 > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.0.0 > > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19120) catalog not properly set for some tables in SQL upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-19120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19120: --- Fix Version/s: 3.1.0 > catalog not properly set for some tables in SQL upgrade scripts > --- > > Key: HIVE-19120 > URL: https://issues.apache.org/jira/browse/HIVE-19120 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.0.0, 3.1.0 > > > A catalog column is added to the PARTITION_EVENTS and NOTIFICATION_LOG but > the upgrade scripts do not include an UPDATE statement to set this to the > default value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19074) Vectorization: Add llap vectorization_div0.q.out Q output file
[ https://issues.apache.org/jira/browse/HIVE-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19074: --- Fix Version/s: (was: 3.0.0) 3.1.0 > Vectorization: Add llap vectorization_div0.q.out Q output file > -- > > Key: HIVE-19074 > URL: https://issues.apache.org/jira/browse/HIVE-19074 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19074.02.patch, HIVE-19074.03.patch, > HIVE-19074.05.patch > > > At some point llap/vectorization_div0.q.out got omitted. > The Q file output is unstable because of missing ORDER BY columns. You must > have ORDER BY on all/critical the columns when there is a LIMIT clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19089) Create/Replicate Allocate write-id event
[ https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431201#comment-16431201 ] Vineet Garg commented on HIVE-19089: Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Create/Replicate Allocate write-id event > > > Key: HIVE-19089 > URL: https://issues.apache.org/jira/browse/HIVE-19089 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, replication > Fix For: 3.1.0 > > Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, > HIVE-19089.03.patch, HIVE-19089.04.patch > > > *EVENT_ALLOCATE_WRITE_ID* > *Source Warehouse:* > * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format > etc. > * Capture this event when allocate a table write ID from the sequence table > by ACID operation. > * Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > * Repl load should read the event from the dump and get the message. > * Validate if source txn ID from the event is there in the source-target txn > ID map. If not there, just noop the event. > * If valid, then Allocate table write ID from sequence table > *Extend listener notify event API to add two new parameter , dbconn and > sqlgenerator to add the events to notification_log table within the same > transaction* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19074) Vectorization: Add llap vectorization_div0.q.out Q output file
[ https://issues.apache.org/jira/browse/HIVE-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431202#comment-16431202 ] Vineet Garg commented on HIVE-19074: Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Vectorization: Add llap vectorization_div0.q.out Q output file > -- > > Key: HIVE-19074 > URL: https://issues.apache.org/jira/browse/HIVE-19074 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-19074.02.patch, HIVE-19074.03.patch, > HIVE-19074.05.patch > > > At some point llap/vectorization_div0.q.out got omitted. > The Q file output is unstable because of missing ORDER BY columns. You must > have ORDER BY on all/critical the columns when there is a LIMIT clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19089) Create/Replicate Allocate write-id event
[ https://issues.apache.org/jira/browse/HIVE-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19089: --- Fix Version/s: (was: 3.0.0) 3.1.0 > Create/Replicate Allocate write-id event > > > Key: HIVE-19089 > URL: https://issues.apache.org/jira/browse/HIVE-19089 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, replication > Fix For: 3.1.0 > > Attachments: HIVE-19089.01.patch, HIVE-19089.02.patch, > HIVE-19089.03.patch, HIVE-19089.04.patch > > > *EVENT_ALLOCATE_WRITE_ID* > *Source Warehouse:* > * Create new event type EVENT_ALLOCATE_WRITE_ID with related message format > etc. > * Capture this event when allocate a table write ID from the sequence table > by ACID operation. > * Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > * Repl load should read the event from the dump and get the message. > * Validate if source txn ID from the event is there in the source-target txn > ID map. If not there, just noop the event. > * If valid, then Allocate table write ID from sequence table > *Extend listener notify event API to add two new parameter , dbconn and > sqlgenerator to add the events to notification_log table within the same > transaction* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18988: --- Fix Version/s: (was: 3.0.0) 3.1.0 > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Find the valid last replication state such that it points to event ID of > open_txn event of oldest on-going txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431203#comment-16431203 ] Vineet Garg commented on HIVE-19054: Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19054.1.patch, HIVE-19054.2.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19054: --- Fix Version/s: (was: 3.0.0) 3.1.0 > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19054.1.patch, HIVE-19054.2.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431205#comment-16431205 ] Vineet Garg commented on HIVE-18988: Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Find the valid last replication state such that it points to event ID of > open_txn event of oldest on-going txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19093) some parts of the Driver runs from the "Background-Pool" in HS2
[
https://issues.apache.org/jira/browse/HIVE-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431204#comment-16431204
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19093:
-
I also noticed the same while working on : HIVE-19133
We are explicitly passing PerfLogger as well to the background thread (as I
have commented on HIVE-19133) which is leading to wrong results and race
conditions.
> some parts of the Driver runs from the "Background-Pool" in HS2
> ---
>
> Key: HIVE-19093
> URL: https://issues.apache.org/jira/browse/HIVE-19093
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
>
> I was looking into perflog results; and the fact that Driver.run open / close
> happens on a different thread caught my eye - this might cause real problems
> since {{Session.get()}} will return an entirely different session in the
> aftermath...most notably there are some lock related calls like: releaseLocks
> {code}
> 2018-04-03T08:36:53,488 DEBUG [2c81c6c1-aa6f-4609-8250-5b1a5360a8ba
> HiveServer2-Handler-Pool: Thread-16242]: log.PerfLogger
> (PerfLogger.java:PerfLogBegin(132)) - from=org.apache.hadoop.hive.ql.Driver>
> 2018-04-03T08:37:21,791 DEBUG [HiveServer2-Background-Pool: Thread-16247]:
> log.PerfLogger (PerfLogger.java:PerfLogEnd(172)) - method=Driver.run start=1522744613488 end=1522744641791 duration=28303
> from=org.apache.hadoop.hive.ql.Driver>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17320) OrcRawRecordMerger.discoverKeyBounds logic can be simplified
[ https://issues.apache.org/jira/browse/HIVE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17320: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > OrcRawRecordMerger.discoverKeyBounds logic can be simplified > > > Key: HIVE-17320 > URL: https://issues.apache.org/jira/browse/HIVE-17320 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 3.1.0 > > > with HIVE-17089 we never have any insert events in the deltas > so if for every split of the base we know min/max key, we can use them to > filter delete events since all files are sorted by RecordIdentifier > So we should be able to create SARG for all delete deltas > the code can be simplified since now min/max key doesn't ever have to be null -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18728) Secure webHCat with SSL
[
https://issues.apache.org/jira/browse/HIVE-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18728:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> Secure webHCat with SSL
> ---
>
> Key: HIVE-18728
> URL: https://issues.apache.org/jira/browse/HIVE-18728
> Project: Hive
> Issue Type: New Feature
> Components: Security
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18728.1.patch, HIVE-18728.2.patch,
> HIVE-18728.3.patch
>
>
> Doc for the issue:
> *Configure WebHCat server to use SSL encryption*
> You can configure WebHCat REST-API to use SSL (Secure Sockets Layer)
> encryption. The following WebHCat properties are added to enable SSL.
> {{templeton.use.ssl}}
> Default value: {{false}}
> Description: Set this to true for using SSL encryption for WebHCat server
> {{templeton.keystore.path}}
> Default value: {{}}
> Description: SSL certificate keystore location for WebHCat server
> {{templeton.keystore.password}}
> Default value: {{}}
> Description: SSL certificate keystore password for WebHCat server
> {{templeton.ssl.protocol.blacklist}}
> Default value: {{SSLv2,SSLv3}}
> Description: SSL Versions to disable for WebHCat server
> {{templeton.host}}
> Default value: {{0.0.0.0}}
> Description: The host address the WebHCat server will listen on.
> *Modifying the {{webhcat-site.xml}} file*
> Configure the following properties in the {{webhcat-site.xml}} file to enable
> SSL encryption on each node where WebHCat is installed:
> {code}
>
>
> templeton.use.ssl
> true
>
>
> templeton.keystore.path
> /path/to/ssl_keystore
>
>
> templeton.keystore.password
> password
>
> {code}
> *Example:* To check status of WebHCat server configured for SSL encryption
> use following command
> {code}
> curl -k 'https://:@:50111/templeton/v1/status'
> {code}
> replace {{}} and {{}} with valid user/password. Replace
> {{}} with your host name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18538) Updating last.repl.id on replica warehouse when dbName specified as a pattern
[ https://issues.apache.org/jira/browse/HIVE-18538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18538: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Updating last.repl.id on replica warehouse when dbName specified as a pattern > - > > Key: HIVE-18538 > URL: https://issues.apache.org/jira/browse/HIVE-18538 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Major > Fix For: 3.1.0 > > > Currently for bootstrap load we dynamically generate the task DAG at runtime > rather than generate the whole task graph at compile / semantic analysis > time. The number of vertex as part of this DAG is controlled via > "hive.repl.approx.max.load.tasks". Till now we only have tested with > replication with one database and providing the destination database name in > the repl load command, the ability to update the last.repl.id is dependent on > this. Given we want to move to a db regex for dumping data we have to relook > at this implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14696) Hive Query Fail with MetaException(message:org.datanucleus.exceptions.NucleusDataStoreException: Size request failed
[
https://issues.apache.org/jira/browse/HIVE-14696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-14696:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> Hive Query Fail with
> MetaException(message:org.datanucleus.exceptions.NucleusDataStoreException:
> Size request failed
> -
>
> Key: HIVE-14696
> URL: https://issues.apache.org/jira/browse/HIVE-14696
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-14696.1.patch
>
>
> We have a customer who is on Hive 0.13 and the queries seem to be failing
> with exception:
> {code}
> 2016-08-30 00:22:58,965 ERROR [main]: metadata.Hive
> (Hive.java:getPartition(1619)) -
> MetaException(message:org.datanucleus.exceptions.NucleusDataStoreException:
> Size request failed : SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE
> THIS.`SD_ID`=? AND THIS.`INTEGER_IDX`>=0)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java:54171)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java:54148)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result.read(ThriftHiveMetastore.java:54079)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition_with_auth(ThriftHiveMetastore.java:1689)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition_with_auth(ThriftHiveMetastore.java:1672)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1003)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
> at com.sun.proxy.$Proxy9.getPartitionWithAuthInfo(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1611)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1565)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:370)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1508)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1275)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1093)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:748)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> It is similar JIRA for Hive 0.13:
> https://issues.apache.org/jira/browse/HIVE-8766
> I suppose it's the similar issues because of both issues related to hive
> metastore performance, can occur when metastore is overloaded and can throw
> different exceptions
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[ https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15444: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > tez.queue.name is invalid after tez job running on CLI > -- > > Key: HIVE-15444 > URL: https://issues.apache.org/jira/browse/HIVE-15444 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 2.2.0 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-15444.1.patch > > > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set tez.queue.name=HQ_OLPS; > hive> set tez.queue.name; > tez.queue.name=HQ_OLPS > hive> insert into abc values(2,2); > Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1 > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (Executing on YARN cluster with App id > application_1481877998678_0011) > -- > VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING > FAILED KILLED > -- > Map 1 .. container SUCCEEDED 1 100 > 0 0 > -- > VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s > -- > Loading data to table default.abc > OK > Time taken: 19.983 seconds > hive> set tez.queue.name; > tez.queue.name is undefined > hive> set hive.execution.engine; > hive.execution.engine=tez -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18279) Incorrect condition in StatsOpimizer
[
https://issues.apache.org/jira/browse/HIVE-18279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18279:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> Incorrect condition in StatsOpimizer
>
>
> Key: HIVE-18279
> URL: https://issues.apache.org/jira/browse/HIVE-18279
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18279.1.patch
>
>
> At the moment {{StatsOpimizer}} has code
> {code}
> if (rowCnt == null) {
> // if rowCnt < 1 than its either empty table or table on which
> stats are not
> // computed We assume the worse and don't attempt to optimize.
> Logger.debug("Table doesn't have up to date stats " +
> tbl.getTableName());
> rowCnt = null;
> }
> {code}
> in method {{private Long getRowCnt()}}. Condition
> {code}
> if (rowCnt == null) {
> {code}
> should be changed to
> {code}
> if (rowCnt == null || rowCnt == 0) {
> {code}
> because 0 value also means that table stats may not be computed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17261) Hive use deprecated ParquetInputSplit constructor which blocked parquet dictionary filter
[ https://issues.apache.org/jira/browse/HIVE-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17261: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Hive use deprecated ParquetInputSplit constructor which blocked parquet > dictionary filter > - > > Key: HIVE-17261 > URL: https://issues.apache.org/jira/browse/HIVE-17261 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.2.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-17261.10.patch, HIVE-17261.11.patch, > HIVE-17261.2.patch, HIVE-17261.3.patch, HIVE-17261.4.patch, > HIVE-17261.5.patch, HIVE-17261.6.patch, HIVE-17261.7.patch, > HIVE-17261.8.patch, HIVE-17261.diff, HIVE-17261.patch > > > Hive use deprecated ParquetInputSplit in > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java#L128] > Please see interface definition in > [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputSplit.java#L80] > Old interface set rowgroupoffset values which will lead to skip dictionary > filter in parquet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed
[
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16041:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> HCatalog doesn't delete temp _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-16041.1.patch, HIVE-16041.2.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table, a
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the
> job failed.
> {quote}
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:45
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:51
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x - yun hdfs 0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r-- 3 yun hdfs 12 2017-02-27 00:57
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r-- 3 yun hdfs 12 2017-02-27 01:28
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19135: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.1.0 > > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18668) Really shade guava in ql
[ https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18668: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Really shade guava in ql > > > Key: HIVE-18668 > URL: https://issues.apache.org/jira/browse/HIVE-18668 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18668.01.patch, HIVE-18668.02.patch > > > After HIVE-15393 a test started to fail in druid; after some investigation it > turned out that ql doesn't shade it's guava artifact at all...because it > shades 'com.google.guava' instead 'com.google.common' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17539) User impersonation failure is not propagated by server as a failure to client
[ https://issues.apache.org/jira/browse/HIVE-17539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17539: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > User impersonation failure is not propagated by server as a failure to client > - > > Key: HIVE-17539 > URL: https://issues.apache.org/jira/browse/HIVE-17539 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Critical > Fix For: 3.1.0 > > > As part of HIVE-17512 we fixed the distCp user impersonation if doAs = false > and configured "hive.distcp.privileged.doAs" is same as the user running > hiveServer. However in the event of not applying the source in patch for > HIVE-17512 and running the corresponding test run in HIVE-17512 with older > code there is impersonation error in the hive server logs, however the driver > returns an "exitValue" of 0 which is wrong, since the copy failed we should > return the error code appropriately. > also since the table creation happens and only data is missing there is a > possibility that the last.repl.id on the table is the latest value with the > data missing and coupled with no error returned to the client this can lead > to serious replication inconsistencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18752) HiveEndPoint: Optimise metastore calls to open transactions batch and allocate write Ids.
[ https://issues.apache.org/jira/browse/HIVE-18752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18752: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > HiveEndPoint: Optimise metastore calls to open transactions batch and > allocate write Ids. > - > > Key: HIVE-18752 > URL: https://issues.apache.org/jira/browse/HIVE-18752 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Metastore >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, Metastore, Streaming > Fix For: 3.1.0 > > > Per table write ID implementation (HIVE-18192) have introduced write ID and > maps it against the txn. > Now, for streaming ingest, we need to open txns batch and then allocate write > id for each txn in the batch which is 2 metastore calls. > This can be optimised to use only one metastore api. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17025) HPL/SQL: hplsql.conn.convert.hiveconn seems to default to false, contrary to docs
[
https://issues.apache.org/jira/browse/HIVE-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17025:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> HPL/SQL: hplsql.conn.convert.hiveconn seems to default to false, contrary to
> docs
> -
>
> Key: HIVE-17025
> URL: https://issues.apache.org/jira/browse/HIVE-17025
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: Carter Shanklin
>Assignee: Dmitry Tolpeko
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-17025.1.patch
>
>
> This bug is part of a series of issues and surprising behavior I encountered
> writing a reporting script that would aggregate values and give rows
> different classifications based on an the aggregate. Addressing some or all
> of these issues would make HPL/SQL more accessible to newcomers.
> Example from the docs is as follows:
> CREATE TABLE dept (
> deptno NUMBER(2,0),
> dname NUMBER(14),
> locVARCHAR2(13),
> CONSTRAINT pk_dept PRIMARY KEY (deptno)
> );
> With this config:
>
>
> hplsql.conn.default
> hiveconn
>
>
> hplsql.conn.hiveconn
> org.apache.hive.jdbc.HiveDriver;jdbc:hive2://
>
>
> I get this error:
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.parse.ParseException:line 2:9 cannot recognize
> input near 'NUMBER' '(' '2' in column type
> With this config:
>
>
> hplsql.conn.default
> hiveconn
>
>
> hplsql.conn.hiveconn
> org.apache.hive.jdbc.HiveDriver;jdbc:hive2://
>
>
> hplsql.conn.convert.hiveconn
> true
>
>
> the example works.
> Version = 3.0.0-SNAPSHOT r71f52d8ad512904b3f2c4f04fe39a33f2834f1f2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18687) Triggers failing to result in event in HA clusters
[ https://issues.apache.org/jira/browse/HIVE-18687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18687: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Triggers failing to result in event in HA clusters > -- > > Key: HIVE-18687 > URL: https://issues.apache.org/jira/browse/HIVE-18687 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.1.0 > > > Triggers in active plan are failing to get picked in some cases, in HA > cluster . In HA environment when the query to activate plan and the test > query (which we expect to get killed by trigger) end up in different > hiveserver2 instances in the same cluster, trigger fails to kick in and kill > the query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18983:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18037) Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x
[ https://issues.apache.org/jira/browse/HIVE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18037: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x > > > Key: HIVE-18037 > URL: https://issues.apache.org/jira/browse/HIVE-18037 > Project: Hive > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18037.001.patch, HIVE-18037.002.patch, > HIVE-18037.003.patch > > > Apache Slider has been migrated to Hadoop-3.x and is referred to as YARN > Service (YARN-4692). Most of the classic Slider features are now going to be > supported in a first-class manner by core YARN. It includes several new > features like a RESTful API. Command line equivalents of classic Slider are > supported by YARN Service as well. > This jira will take care of all changes required to Slider LLAP packaging and > scripts to make it work against Hadoop 3.x. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17484) Bootstrap load should not have metadata only operations for partitions
[
https://issues.apache.org/jira/browse/HIVE-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17484:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> Bootstrap load should not have metadata only operations for partitions
> --
>
> Key: HIVE-17484
> URL: https://issues.apache.org/jira/browse/HIVE-17484
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.1.0
>
>
> Currently since the code path for bootstrap load and import has been
> separated, there is redundant code in the repl boostrap load code path where
> in LoadPartitions looks for {{replicationSpec.isMetadataOnly()}} which should
> never happen. Something similar also happens while bootstrap dump. Look at
> the relevant code paths to clean up code and remove zombie code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18265) desc formatted/extended or show create table can not fully display the result when field or table comment contains tab character
[
https://issues.apache.org/jira/browse/HIVE-18265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18265:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> desc formatted/extended or show create table can not fully display the result
> when field or table comment contains tab character
>
>
> Key: HIVE-18265
> URL: https://issues.apache.org/jira/browse/HIVE-18265
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 1.2.1, 3.0.0
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18265.1.patch, HIVE-18265.patch
>
>
> Here are some examples:
> create table test_comment (id1 string comment 'full_\tname1', id2 string
> comment 'full_\tname2', id3 string comment 'full_\tname3') stored as textfile;
> When execute `show create table test_comment`, we can see the following
> content in the console,
> {quote}
> createtab_stmt
> CREATE TABLE `test_comment`(
> `id1` string COMMENT 'full_
> `id2` string COMMENT 'full_
> `id3` string COMMENT 'full_
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> LOCATION
> 'hdfs://xxx/user/huanghui/warehouse/huanghuitest.db/test_comment'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1513095570')
> {quote}
> And the output of `desc formatted table ` is a little similar,
> {quote}
> col_name data_type comment
> \# col_name data_type comment
> id1 string full_
> id2 string full_
> id3 string full_
> \# Detailed Table Information
> (ignore)...
> {quote}
> When execute `desc extended test_comment`, the problem is more obvious,
> {quote}
> col_name data_type comment
> id1 string full_
> id2 string full_
> id3 string full_
> Detailed Table InformationTable(tableName:test_comment,
> dbName:huanghuitest, owner:huanghui, createTime:1513095570, lastAccessTime:0,
> retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:id1, type:string,
> comment:full_name1), FieldSchema(name:id2, type:string, comment:full_
> {quote}
> *the rest of the content is lost*.
> The content is not really lost, it's just can not display normal. Because
> hive store the result in LazyStruct, and LazyStruct use '\t' as field
> separator:
> {code:java}
> // LazyStruct.java#parse()
> // Go through all bytes in the byte[]
> while (fieldByteEnd <= structByteEnd) {
> if (fieldByteEnd == structByteEnd || bytes[fieldByteEnd] == separator) {
> // Reached the end of a field?
> if (lastColumnTakesRest && fieldId == fields.length - 1) {
> fieldByteEnd = structByteEnd;
> }
> startPosition[fieldId] = fieldByteBegin;
> fieldId++;
> if (fieldId == fields.length || fieldByteEnd == structByteEnd) {
> // All fields have been parsed, or bytes have been parsed.
> // We need to set the startPosition of fields.length to ensure we
> // can use the same formula to calculate the length of each field.
> // For missing fields, their starting positions will all be the
> same,
> // which will make their lengths to be -1 and uncheckedGetField will
> // return these fields as NULLs.
> for (int i = fieldId; i <= fields.length; i++) {
> startPosition[i] = fieldByteEnd + 1;
> }
> break;
> }
> fieldByteBegin = fieldByteEnd + 1;
> fieldByteEnd++;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15944) The order of cols is error in ColumnPrunerReduceSinkProc because of sort operator
[ https://issues.apache.org/jira/browse/HIVE-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15944: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > The order of cols is error in ColumnPrunerReduceSinkProc because of sort > operator > - > > Key: HIVE-15944 > URL: https://issues.apache.org/jira/browse/HIVE-15944 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.1.0, 2.2.0 >Reporter: wan kun >Assignee: wan kun >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-15944-branch-1.1.patch, HIVE-15944.1.patch, > HIVE-15944.2-branch-1.1.1.path.erroroutput, HIVE-15944.3.patch, > HIVE-15944.4-branch-1.1.1.patch, HIVE-15944.4.patch, HIVE-15944.5.patch, > HIVE-15944.6.patch, HIVE-15944.7.patch, HIVE-15944.8.patch, HIVE-15944.patch, > STAGE_DEPENDENCIES > > Original Estimate: 168h > Remaining Estimate: 168h > > If one sql have two job. > job 1:The order of cols is updated in ColumnPrunerReduceSinkProc because of > sort operator. > job 2 will read error in map operation because the cols order is old. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18476) copy hdfs ACL's as part of replication
[ https://issues.apache.org/jira/browse/HIVE-18476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18476: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > copy hdfs ACL's as part of replication > -- > > Key: HIVE-18476 > URL: https://issues.apache.org/jira/browse/HIVE-18476 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Major > Fix For: 3.1.0 > > > with improvements to HDFS ACL's in hadoop 3.0, hive should, as part of > replication also copy over the ACL's when copying files to target warehouse. > this would also mean setting the correct owner name and group name > so setOwner + setAcl has to be done on the files copied. > reference: > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-13697) ListBucketing feature does not support uppercase string.
[
https://issues.apache.org/jira/browse/HIVE-13697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-13697:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> ListBucketing feature does not support uppercase string.
>
>
> Key: HIVE-13697
> URL: https://issues.apache.org/jira/browse/HIVE-13697
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 1.2.1
> Environment: 1.2.1
>Reporter: Hao Zhu
>Assignee: Oleksiy Sayankin
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: HIVE-13697.1.patch
>
>
> This is the feature:
> https://cwiki.apache.org/confluence/display/Hive/ListBucketing
> 1. Good example:
> {code}
> CREATE TABLE testskew (id INT, a STRING)
> SKEWED BY (a) ON ('abc', 'xyz') STORED AS DIRECTORIES;
> set hive.mapred.supports.subdirectories=true;
> set mapred.input.dir.recursive=true;
> INSERT OVERWRITE TABLE testskew
> SELECT 123,'abc' FROM dual
> union all
> SELECT 123,'xyz' FROM dual
> union all
> SELECT 123,'others' FROM dual;
> {code}
> {code}
> # hadoop fs -ls /user/hive/warehouse/testskew
> Found 3 items
> drwxrwxrwx - mapr mapr 1 2016-05-05 14:56
> /user/hive/warehouse/testskew/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
> drwxrwxrwx - mapr mapr 1 2016-05-05 14:56
> /user/hive/warehouse/testskew/a=abc
> drwxrwxrwx - mapr mapr 1 2016-05-05 14:56
> /user/hive/warehouse/testskew/a=xyz
> {code}
> This is good, because both "abc" and "xyz" directories got created.
> 2. Bad example -- This is the issue
> {code}
> CREATE TABLE testskew2 (id INT, a STRING)
> SKEWED BY (a) ON ('aus', 'US') STORED AS DIRECTORIES;
> set hive.mapred.supports.subdirectories=true;
> set mapred.input.dir.recursive=true;
> INSERT OVERWRITE TABLE testskew2
> SELECT 123, 'aus' FROM dual
> union all
> SELECT 123, 'US' FROM dual
> union all
> SELECT 123, 'others' FROM dual;
> {code}
> You can see, only "aus" directory got created...
> {code}
> # hadoop fs -ls /user/hive/warehouse/testskew2
> Found 2 items
> drwxrwxrwx - mapr mapr 1 2016-05-05 15:11
> /user/hive/warehouse/testskew2/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
> drwxrwxrwx - mapr mapr 1 2016-05-05 15:11
> /user/hive/warehouse/testskew2/a=aus
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14981) Eliminate unnecessary MapJoin restriction in HIVE-11394
[ https://issues.apache.org/jira/browse/HIVE-14981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-14981: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Eliminate unnecessary MapJoin restriction in HIVE-11394 > --- > > Key: HIVE-14981 > URL: https://issues.apache.org/jira/browse/HIVE-14981 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-14981.01.patch, HIVE-14981.02.patch > > > No Empty Key restriction for Native Vector MapJoin added with HIVE-11394 is > unnecessary. It caused Llap orc_llap.q test to timeout on Hive QA because > regular VectorMapJoinOperator is too slow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18872) Projection is not pushed properly when query involves multiple tables
[ https://issues.apache.org/jira/browse/HIVE-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18872: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Projection is not pushed properly when query involves multiple tables > - > > Key: HIVE-18872 > URL: https://issues.apache.org/jira/browse/HIVE-18872 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18872.patch > > > Projections are not pushed down properly during getSplit() when the query > involves multiple tables although they are getting properly pushed during > getRecordReader when the task is working on the split. Due to this, storage > handler relying on projections for building query while generating input > splits are not working. > here, in below case, due to bug we will be pushing ID2 for both the aliases > "A" and "B" during addSplitsForGroup instead of pushing DB for alias "A" and > ID2 only for alias "B". > SELECT A.ID, a.db, B.ID2 from joinTable3 A join joinTable4 B on A.ID = B.ID > WHERE A.ID=10; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17579) repl load without providing the database name in the command fails.
[
https://issues.apache.org/jira/browse/HIVE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17579:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> repl load without providing the database name in the command fails.
> ---
>
> Key: HIVE-17579
> URL: https://issues.apache.org/jira/browse/HIVE-17579
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Fix For: 3.1.0
>
>
> repl dump [databasename] = > [hdfs location]
> if we run {{repl load [hdfs location]}} this fails. it should pick the
> database name from the metadata file in this case but leads to a HiveException
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[
https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-14304:
---
Fix Version/s: (was: 3.0.0)
3.1.0
Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please
update the JIRA if you would like to get your patch in 3.0.0.
> Beeline command will fail when entireLineAsCommand set to true
> --
>
> Key: HIVE-14304
> URL: https://issues.apache.org/jira/browse/HIVE-14304
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-14304.1.patch
>
>
> Use beeline
> {code}
> beeline --entireLineAsCommand=true
> {code}
> show tables fail:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> show tables;
> Error: Error while compiling statement: FAILED: ParseException line 1:11
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> {code}
> We should remove the trailing semi-colon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15222) replace org.json usage in ExplainTask/TezTask related classes with some alternative
[ https://issues.apache.org/jira/browse/HIVE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15222: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > replace org.json usage in ExplainTask/TezTask related classes with some > alternative > --- > > Key: HIVE-15222 > URL: https://issues.apache.org/jira/browse/HIVE-15222 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-15222.1.patch, HIVE-15222.2.patch, > HIVE-15222.3.patch > > > Replace org.json usage in these classes. > It seems to me that json is probably only used to write some information - > but the application never reads it back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15945) Remove debug parameter in HADOOP_OPTS environment when start a new job local.
[ https://issues.apache.org/jira/browse/HIVE-15945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15945: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Remove debug parameter in HADOOP_OPTS environment when start a new job local. > - > > Key: HIVE-15945 > URL: https://issues.apache.org/jira/browse/HIVE-15945 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.2.0 >Reporter: wan kun >Assignee: wan kun >Priority: Minor > Labels: patch > Fix For: 3.1.0 > > Attachments: HIVE-15945.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > When hive start a new job in child VM,the debug parameter will be defined > twice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17227) Incremental replication load should creates tasks in execution phase rather than semantic phase
[ https://issues.apache.org/jira/browse/HIVE-17227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17227: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Incremental replication load should creates tasks in execution phase rather > than semantic phase > > > Key: HIVE-17227 > URL: https://issues.apache.org/jira/browse/HIVE-17227 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Fix For: 3.1.0 > > > as we did for bootstrap replication load in HIVE-16896 we should use a > mechanism to dynamically create dag graph for incremental replication as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16894) Multi-threaded execution of bootstrap dump of tables / functions
[ https://issues.apache.org/jira/browse/HIVE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-16894: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Multi-threaded execution of bootstrap dump of tables / functions > > > Key: HIVE-16894 > URL: https://issues.apache.org/jira/browse/HIVE-16894 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Fix For: 3.1.0 > > > after completing HIVE-16893 the bootstrap process will dump single table at a > time and hence will be very time consuming while not optimally utilizing the > available resources. Since there is no dependency between dumps of various > tables we should be able to do this in parallel. > Bootstrap dump at db level does : > * boostrap of all tables (scope of current jira) > ** boostrap of all partitions in a table. > * boostrap of all functions (scope of current jira) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18702) INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
[
https://issues.apache.org/jira/browse/HIVE-18702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18702:
---
Fix Version/s: (was: 3.0.0)
3.1.0
> INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
> ---
>
> Key: HIVE-18702
> URL: https://issues.apache.org/jira/browse/HIVE-18702
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 2.4.0, 3.1.0
>
> Attachments: HIVE-18702.1.patch, HIVE-18702.2.patch
>
>
> Enable Hive on TEZ. (MR works fine).
> *STEP 1. Create test data*
> {code}
> nano /home/test/users.txt
> {code}
> Add to file:
> {code}
> Peter,34
> John,25
> Mary,28
> {code}
> {code}
> hadoop fs -mkdir /bug
> hadoop fs -copyFromLocal /home/test/users.txt /bug
> hadoop fs -ls /bug
> {code}
> *EXPECTED RESULT:*
> {code}
> Found 2 items
>
> -rwxr-xr-x 3 root root 25 2015-10-15 16:11 /bug/users.txt
> {code}
> *STEP 2. Upload data to hive*
> {code}
> create external table bug(name string, age int) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',' LINES TERMINATED BY '\n' LOCATION '/bug';
> select * from bug;
> {code}
> *EXPECTED RESULT:*
> {code}
> OK
> Peter 34
> John25
> Mary28
> {code}
> {code}
> create external table bug1(name string, age int) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',' LINES TERMINATED BY '\n' LOCATION '/bug1';
> insert overwrite table bug select * from bug1;
> select * from bug;
> {code}
> *EXPECTED RESULT:*
> {code}
> OK
> Time taken: 0.097 seconds
> {code}
> *ACTUAL RESULT:*
> {code}
> hive> select * from bug;
> OK
> Peter 34
> John 25
> Mary 28
> Time taken: 0.198 seconds, Fetched: 3 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15121) Last MR job in Hive should be able to write to a different scratch directory
[ https://issues.apache.org/jira/browse/HIVE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15121: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Last MR job in Hive should be able to write to a different scratch directory > > > Key: HIVE-15121 > URL: https://issues.apache.org/jira/browse/HIVE-15121 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-15121.1.patch, HIVE-15121.2.patch, > HIVE-15121.3.patch, HIVE-15121.WIP.1.patch, HIVE-15121.WIP.2.patch, > HIVE-15121.WIP.patch, HIVE-15121.patch > > > Hive should be able to configure all intermediate MR jobs to write to HDFS, > but the final MR job to write to S3. > This will be useful for implementing parallel renames on S3. The idea is that > for a multi-job query, all intermediate MR jobs write to HDFS, and then the > final job writes to S3. Writing to HDFS should be faster than writing to S3, > so it makes more sense to write intermediate data to HDFS. > The advantage is that any copying of data that needs to be done from the > scratch directory to the final table directory can be done server-side, > within the blobstore. The MoveTask simply renames data from the scratch > directory to the final table location, which should translate to a > server-side COPY request. This way HiveServer2 doesn't have to actually copy > any data, it just tells the blobstore to do all the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14925) MSCK repair table hang while running with multi threading enabled
[ https://issues.apache.org/jira/browse/HIVE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-14925: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > MSCK repair table hang while running with multi threading enabled > - > > Key: HIVE-14925 > URL: https://issues.apache.org/jira/browse/HIVE-14925 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 2.2.0 >Reporter: Ratheesh Kamoor >Assignee: Ratheesh Kamoor >Priority: Critical > Fix For: 3.1.0 > > Attachments: HIVE-14925.patch > > > MSCK REPAIR TABLE hanging while running with multi-threading enabled > (default). I think it is because of a major design flaw in how thread pool > implemented in HiveMetaSoreChecker class / checkPartitionDirs method. This > method has a thread pool which register Callable but callable makes a > recursive call to checkPartitionDirs method again. This code will hang when > number of directories is more than thread pool size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18857) Store default value text instead of default value expression in metastore
[ https://issues.apache.org/jira/browse/HIVE-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18857: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > Store default value text instead of default value expression in metastore > - > > Key: HIVE-18857 > URL: https://issues.apache.org/jira/browse/HIVE-18857 > Project: Hive > Issue Type: Improvement > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18857.1.patch, HIVE-18857.2.patch, > HIVE-18857.3.patch > > > Currently for default value an expression is generated and serialized to > store in metastore. It should be improved to rather serialize the default > value itself instead of expression and store that in metastore. This will > have the following benefits: > * It will make metastore schema upgrade safe. e.g. if a UDF function name is > changed hive wouldn't be able to parse back the expression for this UDF which > was serialized in earlier version. > * It will make metastore schema for default constraint hive agnostic. Other > databases would be able to use the value as it is. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15223) replace org.json usage in EximUtil with some alternative
[ https://issues.apache.org/jira/browse/HIVE-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15223: --- Fix Version/s: (was: 3.0.0) 3.1.0 Deferring this to 3.1.0 since the branch for 3.0.0 has been cut off. Please update the JIRA if you would like to get your patch in 3.0.0. > replace org.json usage in EximUtil with some alternative > > > Key: HIVE-15223 > URL: https://issues.apache.org/jira/browse/HIVE-15223 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-15223.1.patch > > > The metadata is stored in json format...which changed lately with the advent > of replication v2. > I think jackson would be nice to have here - it could possibly aid to make > this Metadata reading / writing more resilient against future serialization > issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
