[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034208#comment-17034208 ] Ivan Suller commented on HIVE-21000: [~kgyrtkirk] [~q.xu] Accumulo 2 has been release and a ticket has been opened to upgrade to it: HIVE-0. I started working on it, but was way more complicated then I anticipated and had way less time for it so I could never finish. If someone could pick it up that would be a first step for Thrift upgrade. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22763) 0 is accepted in 12-hour format during timestamp cast
[
https://issues.apache.org/jira/browse/HIVE-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034203#comment-17034203
]
Hive QA commented on HIVE-22763:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993074/HIVE-22763.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17990 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_external_acid]
(batchId=106)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testHiveRefreshOnConfChange
(batchId=338)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20552/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20552/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20552/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993074 - PreCommit-HIVE-Build
> 0 is accepted in 12-hour format during timestamp cast
> -
>
> Key: HIVE-22763
> URL: https://issues.apache.org/jira/browse/HIVE-22763
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22763.01.patch, HIVE-22763.01.patch,
> HIVE-22763.01.patch, HIVE-22763.01.patch
>
>
> Having a timestamp string in 12-hour format can be parsed if the hour is 0,
> however, based on the [design
> document|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit],
> it should be rejected.
> h3. How to reproduce
> Run {code}select cast("2020-01-01 0 am 00" as timestamp format "-mm-dd
> hh12 p.m. ss"){code}
> It shouldn' t be parsed, as the hour component is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034185#comment-17034185 ] László Bodor commented on HIVE-22864: - +1 > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Attachment: (was: HIVE-22864.01.patch) > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Attachment: HIVE-22864.01.patch > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22763) 0 is accepted in 12-hour format during timestamp cast
[
https://issues.apache.org/jira/browse/HIVE-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034165#comment-17034165
]
Hive QA commented on HIVE-22763:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20552/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20552/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> 0 is accepted in 12-hour format during timestamp cast
> -
>
> Key: HIVE-22763
> URL: https://issues.apache.org/jira/browse/HIVE-22763
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22763.01.patch, HIVE-22763.01.patch,
> HIVE-22763.01.patch, HIVE-22763.01.patch
>
>
> Having a timestamp string in 12-hour format can be parsed if the hour is 0,
> however, based on the [design
> document|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit],
> it should be rejected.
> h3. How to reproduce
> Run {code}select cast("2020-01-01 0 am 00" as timestamp format "-mm-dd
> hh12 p.m. ss"){code}
> It shouldn' t be parsed, as the hour component is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-22858) HMS broken with mysql
[
https://issues.apache.org/jira/browse/HIVE-22858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan resolved HIVE-22858.
-
Resolution: Duplicate
> HMS broken with mysql
> -
>
> Key: HIVE-22858
> URL: https://issues.apache.org/jira/browse/HIVE-22858
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Using commit 7bb1d1edfcba558958265ec47245bc529eaee2d8 (Jan 27) apache master.
> Encountered the following exception when creating new database in hive (mysql
> for HMS).
> https://issues.apache.org/jira/browse/HIVE-22663 may be related to this.
>
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near '"NEXT_TXN_ID" for update' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:599)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:555)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19779)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19764)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:565)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
>
[jira] [Commented] (HIVE-22858) HMS broken with mysql
[
https://issues.apache.org/jira/browse/HIVE-22858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034160#comment-17034160
]
Rajesh Balamohan commented on HIVE-22858:
-
Thanks [~zchovan] . Closing this ticket.
> HMS broken with mysql
> -
>
> Key: HIVE-22858
> URL: https://issues.apache.org/jira/browse/HIVE-22858
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Using commit 7bb1d1edfcba558958265ec47245bc529eaee2d8 (Jan 27) apache master.
> Encountered the following exception when creating new database in hive (mysql
> for HMS).
> https://issues.apache.org/jira/browse/HIVE-22663 may be related to this.
>
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near '"NEXT_TXN_ID" for update' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:599)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:555)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19779)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19764)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:565)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_112]
> at
>
[jira] [Commented] (HIVE-10362) Support Type check/conversion in dynamic partition column
[
https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034157#comment-17034157
]
Hive QA commented on HIVE-10362:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993073/HIVE-10362.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17991 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20551/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20551/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20551/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993073 - PreCommit-HIVE-Build
> Support Type check/conversion in dynamic partition column
> -
>
> Key: HIVE-10362
> URL: https://issues.apache.org/jira/browse/HIVE-10362
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, Types
>Affects Versions: 1.0.0, 3.0.0, 4.0.0
>Reporter: Chaoyu Tang
>Assignee: Karen Coppage
>Priority: Major
> Labels: breaking_change
> Fix For: 4.0.0
>
> Attachments: HIVE-10362.01.patch, HIVE-10362.02.patch,
> HIVE-10362.03.patch, HIVE-10362.04.patch, HIVE-10362.04.patch,
> HIVE-10362.06.patch, HIVE-10362.07.patch, HIVE-10362.07.patch,
> HIVE-10362.08.patch, HIVE-10362.09.patch, HIVE-10362.10.patch,
> HIVE-10362.11.patch, HIVE-10362.12.patch, HIVE-10362.12.patch,
> HIVE-10362.12.patch, HIVE-10362.12.patch, HIVE-10362.12.patch
>
>
> There are quite a lot of issues associated with the non-noramlized or
> type-mismatched values for partition column. Hive has many ways to introduce
> such problematic data. HIVE-10307 mainly provides the support to type
> check/convert/normalize the partition column value in static partition
> specification. This JIRA tries to deal with the partition column type in
> dynamic partition insert. Currently any data can be inserted as a partition
> column value as long as it is quoted as a string. For example,
> create table dynparttypechecknum (key int, value string) partitioned by (part
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1'
> from src limit 1;
> show partitions dynparttypechecknum;
> --
> part=1
> The partition column value is non-normalized int 1. It causes some
> unnecessary problems such as integer partition column JDO filter pushdown
> (see HIVE-6052) and others like HIVE-10210.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22844) Validate cm configs, add retries in fs apis for cm
[ https://issues.apache.org/jira/browse/HIVE-22844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22844: --- Attachment: HIVE-22844.patch Status: Patch Available (was: In Progress) > Validate cm configs, add retries in fs apis for cm > -- > > Key: HIVE-22844 > URL: https://issues.apache.org/jira/browse/HIVE-22844 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22844.patch, HIVE-22844.patch, HIVE-22844.patch, > HIVE-22844.patch, HIVE-22844.patch > > Time Spent: 10m > Remaining Estimate: 0h > > # Retry create cm root logic > # Rename encryptionZones to cmRootLocations to be more accurate > # Check cmRootEncrypted.isAbsolute() first before we go for creating anything > # Validate fallbackNonEncryptedCmRootDir if it's really not encrypted > # Refactor deleteTableData logic -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22844) Validate cm configs, add retries in fs apis for cm
[ https://issues.apache.org/jira/browse/HIVE-22844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22844: --- Status: In Progress (was: Patch Available) > Validate cm configs, add retries in fs apis for cm > -- > > Key: HIVE-22844 > URL: https://issues.apache.org/jira/browse/HIVE-22844 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22844.patch, HIVE-22844.patch, HIVE-22844.patch, > HIVE-22844.patch, HIVE-22844.patch > > Time Spent: 10m > Remaining Estimate: 0h > > # Retry create cm root logic > # Rename encryptionZones to cmRootLocations to be more accurate > # Check cmRootEncrypted.isAbsolute() first before we go for creating anything > # Validate fallbackNonEncryptedCmRootDir if it's really not encrypted > # Refactor deleteTableData logic -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034143#comment-17034143
]
Shivam edited comment on HIVE-22771 at 2/11/20 5:38 AM:
Any comments? Do you think this patch is okay to go? or I should try
generating a string hashId?
was (Author: shivam-mohan):
[~mithun] [~dvoros] Any comments?. Do you think this patch is okay to go? or I
should try generating a string hashId?
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch, HIVE-22771.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d\\.?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22771) Partition location incorrectly formed in FileOutputCommitterContainer
[
https://issues.apache.org/jira/browse/HIVE-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034143#comment-17034143
]
Shivam commented on HIVE-22771:
---
[~mithun] [~dvoros] Any comments?. Do you think this patch is okay to go? or I
should try generating a string hashId?
> Partition location incorrectly formed in FileOutputCommitterContainer
> -
>
> Key: HIVE-22771
> URL: https://issues.apache.org/jira/browse/HIVE-22771
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.1
>Reporter: Shivam
>Assignee: Shivam
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-22771.2.patch, HIVE-22771.3.patch, HIVE-22771.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Class _HCatOutputFormat_ in package _org.apache.hive.hcatalog.mapreduce_ uses
> function _setOutput_ to generate _idHash_ using below statement:
> *+In file org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.java+*
> *line 116: idHash = String.valueOf(Math.random());*
> The output of idHash can be similar to values like this : 7.145347157239135E-4
>
> And, in class _FileOutputCommitterContainer_ in package
> _org.apache.hive.hcatalog.mapreduce;_
> Uses below statement to compute final partition path:
> +*In org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java*+
> *line 366: String finalLocn = jobLocation.replaceAll(Path.SEPARATOR +
> SCRATCH_DIR_NAME + "{color:#ff}\\d\\.?
> d+"{color},"");*
> *line 367: partPath = new Path(finalLocn);*
>
> Regex used here is incorrect, since it will only remove integers after the
> *SCRATCH_DIR_NAME,* and hence will append 'E-4' (for the above example) in
> the final partition location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-10362) Support Type check/conversion in dynamic partition column
[
https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034142#comment-17034142
]
Hive QA commented on HIVE-10362:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 353 unchanged - 1
fixed = 354 total (was 354) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20551/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20551/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20551/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support Type check/conversion in dynamic partition column
> -
>
> Key: HIVE-10362
> URL: https://issues.apache.org/jira/browse/HIVE-10362
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, Types
>Affects Versions: 1.0.0, 3.0.0, 4.0.0
>Reporter: Chaoyu Tang
>Assignee: Karen Coppage
>Priority: Major
> Labels: breaking_change
> Fix For: 4.0.0
>
> Attachments: HIVE-10362.01.patch, HIVE-10362.02.patch,
> HIVE-10362.03.patch, HIVE-10362.04.patch, HIVE-10362.04.patch,
> HIVE-10362.06.patch, HIVE-10362.07.patch, HIVE-10362.07.patch,
> HIVE-10362.08.patch, HIVE-10362.09.patch, HIVE-10362.10.patch,
> HIVE-10362.11.patch, HIVE-10362.12.patch, HIVE-10362.12.patch,
> HIVE-10362.12.patch, HIVE-10362.12.patch, HIVE-10362.12.patch
>
>
> There are quite a lot of issues associated with the non-noramlized or
> type-mismatched values for partition column. Hive has many ways to introduce
[jira] [Commented] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034134#comment-17034134
]
Hive QA commented on HIVE-22841:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993070/HIVE-22841.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17993 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20550/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20550/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20550/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993070 - PreCommit-HIVE-Build
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034127#comment-17034127
]
Rajesh Balamohan commented on HIVE-22850:
-
Attaching .3 version of patch.
1. SQL queries are surrounded with \".
2. Attached jumptableInfo listing the transition details in this jira.
3. In {{TxnHandler::checkLock}}, for DBs and lots being requested for are in
SHARED_READ mode, we can cut down the mutex call. This is added in .3 patch.
4. In {{TxnHandler::enqueueLockWithRetry}}, some of the computations can be
outside of the NEXT_LOCK_ID statement. This is added in .3 patch.
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, HIVE-22850.2.patch,
> HIVE-22850.3.patch, Screenshot 2020-02-07 at 4.14.51 AM.jpg, jumpTableInfo.png
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22850:
Attachment: HIVE-22850.3.patch
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, HIVE-22850.2.patch,
> HIVE-22850.3.patch, Screenshot 2020-02-07 at 4.14.51 AM.jpg, jumpTableInfo.png
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22850) Optimise lock acquisition in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22850:
Attachment: jumpTableInfo.png
> Optimise lock acquisition in TxnHandler
> ---
>
> Key: HIVE-22850
> URL: https://issues.apache.org/jira/browse/HIVE-22850
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22850.1.patch, HIVE-22850.2.patch,
> HIVE-22850.3.patch, Screenshot 2020-02-07 at 4.14.51 AM.jpg, jumpTableInfo.png
>
>
> With concurrent queries, time taken for lock acquisition increases
> substantially. As part of lock acquisition in the query,
> {{TxnHandler::checkLock}} gets invoked. This involves getting a mutex and
> compare the locks being requested for, with that of existing locks in
> {{HIVE_LOCKS}} table.
> With concurrent queries, time taken to do this check increase and this
> significantly increases the time taken for getting mutex for other threads
> (due to select for update). In a synthetic workload, it was in the order of
> 10+ seconds. This codepath can be optimized when all lock requests are
> SHARED_READ.
>
>
> !Screenshot 2020-02-07 at 4.14.51 AM.jpg|width=743,height=348!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034120#comment-17034120
]
Hive QA commented on HIVE-22841:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 32 unchanged -
7 fixed = 32 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} jdbc: The patch generated 1 new + 37 unchanged - 0
fixed = 38 total (was 37) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20550/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20550/yetus/diff-checkstyle-jdbc.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20550/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: service jdbc itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20550/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
>
[jira] [Commented] (HIVE-22824) JoinProjectTranspose rule should skip Projects containing windowing expression
[
https://issues.apache.org/jira/browse/HIVE-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034109#comment-17034109
]
Hive QA commented on HIVE-22824:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12992798/HIVE-22824.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17975 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=160)
[unionDistinct_1.q,table_nonprintable.q,file_with_header_footer_aggregation.q,vector_offset_limit.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q,orc_merge3.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20549/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20549/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20549/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12992798 - PreCommit-HIVE-Build
> JoinProjectTranspose rule should skip Projects containing windowing expression
> --
>
> Key: HIVE-22824
> URL: https://issues.apache.org/jira/browse/HIVE-22824
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22824.1.patch, HIVE-22824.2.patch,
> HIVE-22824.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Otherwise this rule could end up creating plan with windowing expression
> within join condition which hive doesn't know how to process.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22865) Include data in replication staging directory
[ https://issues.apache.org/jira/browse/HIVE-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA reassigned HIVE-22865: - > Include data in replication staging directory > - > > Key: HIVE-22865 > URL: https://issues.apache.org/jira/browse/HIVE-22865 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034096#comment-17034096 ] Shubham Chaurasia commented on HIVE-20312: -- Thanks [~jdere] > Allow arrow clients to use their own BufferAllocator with > LlapOutputFormatService > - > > Key: HIVE-20312 > URL: https://issues.apache.org/jira/browse/HIVE-20312 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, > HIVE-20312.3.patch > > > Clients should be able to provide their own BufferAllocator to > LlapBaseInputFormat if allocator operations depend on client-side logic. For > example, clients may want to manage the allocator hierarchy per client-side > task, thread, etc.. > Currently the client is forced to use one global RootAllocator per process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22824) JoinProjectTranspose rule should skip Projects containing windowing expression
[
https://issues.apache.org/jira/browse/HIVE-22824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034091#comment-17034091
]
Hive QA commented on HIVE-22824:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 492 unchanged - 2
fixed = 492 total (was 494) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
3s{color} | {color:red} ql generated 1 new + 1531 unchanged - 1 fixed = 1532
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(ASTNode,
SemanticAnalyzer$PlannerContext) At CalcitePlanner.java:is not thrown in
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(ASTNode,
SemanticAnalyzer$PlannerContext) At CalcitePlanner.java:[line 563] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20549/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20549/yetus/new-findbugs-ql.html
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20549/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JoinProjectTranspose rule should skip Projects containing windowing expression
> --
>
> Key: HIVE-22824
> URL: https://issues.apache.org/jira/browse/HIVE-22824
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22824.1.patch, HIVE-22824.2.patch,
> HIVE-22824.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Otherwise this rule could end up creating plan with windowing expression
> within join condition which hive doesn't know how to process.
--
This
[jira] [Commented] (HIVE-22825) Reduce directory lookup cost for acid tables
[ https://issues.apache.org/jira/browse/HIVE-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034078#comment-17034078 ] Ashutosh Chauhan commented on HIVE-22825: - +1 > Reduce directory lookup cost for acid tables > > > Key: HIVE-22825 > URL: https://issues.apache.org/jira/browse/HIVE-22825 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22825.1.patch, HIVE-22825.2.patch, > HIVE-22825.3.patch, HIVE-22825.4.patch, HIVE-22825.5.patch > > > With objectstores, directory lookup costs are expensive. For acid tables, it > would be good to have a directory cache to reduce number of lookup calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[
https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034075#comment-17034075
]
Hive QA commented on HIVE-22453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12992972/HIVE-22453.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17990 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
(batchId=200)
org.apache.hadoop.hive.ql.TestTxnCommands.testParallelInsertAnalyzeStats
(batchId=359)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20548/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20548/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20548/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12992972 - PreCommit-HIVE-Build
> Describe table unnecessarily fetches partitions
> ---
>
> Key: HIVE-22453
> URL: https://issues.apache.org/jira/browse/HIVE-22453
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22453.2.patch, HIVE-22453.2.patch, HIVE-22453.patch
>
>
> The simple describe table command without EXTENDED and FORMATTED (i.e.,
> DESCRIBE table_name) fetches all partitions when no partition is specified,
> although it does not display partition statistics in nature.
> The command should not fetch partitions since it can take a long time for a
> large amount of partitions.
> For instance, in our environment, the command takes around 8 seconds for a
> table with 8760 (24 * 365) partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[
https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034054#comment-17034054
]
Hive QA commented on HIVE-22453:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20548/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20548/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Describe table unnecessarily fetches partitions
> ---
>
> Key: HIVE-22453
> URL: https://issues.apache.org/jira/browse/HIVE-22453
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22453.2.patch, HIVE-22453.2.patch, HIVE-22453.patch
>
>
> The simple describe table command without EXTENDED and FORMATTED (i.e.,
> DESCRIBE table_name) fetches all partitions when no partition is specified,
> although it does not display partition statistics in nature.
> The command should not fetch partitions since it can take a long time for a
> large amount of partitions.
> For instance, in our environment, the command takes around 8 seconds for a
> table with 8760 (24 * 365) partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22263) MV has distinct on columns and query has count(distinct) on one of the columns, we do not trigger rewriting
[ https://issues.apache.org/jira/browse/HIVE-22263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22263: --- Priority: Critical (was: Major) > MV has distinct on columns and query has count(distinct) on one of the > columns, we do not trigger rewriting > --- > > Key: HIVE-22263 > URL: https://issues.apache.org/jira/browse/HIVE-22263 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Priority: Critical > Attachments: count-distinct.sql, count-distinct2.sql > > > Count distinct issues with materialized views. Two scripts attached > 1) > create materialized view base_aview stored as orc as select distinct c1 c1, > c2 c2 from base; > explain extended select count(distinct c1) from base group by c2 ; > 2) > create materialized view base_aview stored as orc as SELECT c1 c1, c2 c2, > sum(c2) FROM base group by 1,2; > explain extended select count(distinct c1) from base group by c2; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22265) Ordinals in view are not being picked up in materialized view
[ https://issues.apache.org/jira/browse/HIVE-22265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22265: --- Priority: Critical (was: Minor) > Ordinals in view are not being picked up in materialized view > - > > Key: HIVE-22265 > URL: https://issues.apache.org/jira/browse/HIVE-22265 > Project: Hive > Issue Type: Sub-task > Components: CBO, Materialized views >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Priority: Critical > Attachments: lv-exp-unalias.sql > > > There is a conf which allows ordinals to be used for group by, i.e. > hive.groupby.position.alias > This isn't being picked up by materialized views when set in a view. > Workaround is to not use ordinals. Script is attached. > Example: > create view campaigns.campaign_data_lview_bad as SELECT platform, > platform_version, currency, sum(amount) as sum_amount, sum(duration) as > sum_duration, count(user_ id) count_user_id, min(amount) min_amount, > max(amount) max_amount, year , month FROM `campaigns`.`campaign_data` GROUP > BY 1, 2, 3, 9, 10; > create materialized view aview620_bad stored as orc as select platform, > platform_version, currency, sum_amount, sum_duration, count_user_id, > min_amount, max_amoun t, year, month from > `campaigns`.`campaign_data_lview_bad`; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
[
https://issues.apache.org/jira/browse/HIVE-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034045#comment-17034045
]
Hive QA commented on HIVE-22828:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993068/HIVE-22828.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 17991 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1]
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal64_mul_intcolumn]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_10]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_17]
(batchId=101)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_3]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_1]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal64_case_when]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal64_mul_intcolumn]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reuse_scratchcols]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_10]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_13]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_17]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_1]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_3]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=197)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_1]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=154)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_10]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17]
(batchId=156)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_1]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_3]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
(batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=142)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query36]
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70]
(batchId=305)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query86]
(batchId=305)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query36]
(batchId=303)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query36]
(batchId=303)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query70]
[jira] [Commented] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
[
https://issues.apache.org/jira/browse/HIVE-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034020#comment-17034020
]
Hive QA commented on HIVE-22828:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20546/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20546/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
> -
>
> Key: HIVE-22828
> URL: https://issues.apache.org/jira/browse/HIVE-22828
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal Vijayaraghavan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22828.1.patch, HIVE-22828.3.patch
>
>
> {code}
> select
> sum(case when (ss_item_sk=1) then ss_sales_price else null end),
> sum(case when (ss_item_sk=2) then ss_sales_price else ss_sales_price+1 end),
> sum(case when (ss_item_sk=2) then 1.0BD+ss_sales_price else null end)
> from store_sales where ss_sold_date_sk % = 1;
> {code}
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal.aggregateInputSelection(VectorUDAFSumDecimal.java:113)
> at
>
[jira] [Updated] (HIVE-22825) Reduce directory lookup cost for acid tables
[ https://issues.apache.org/jira/browse/HIVE-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22825: Attachment: HIVE-22825.5.patch > Reduce directory lookup cost for acid tables > > > Key: HIVE-22825 > URL: https://issues.apache.org/jira/browse/HIVE-22825 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22825.1.patch, HIVE-22825.2.patch, > HIVE-22825.3.patch, HIVE-22825.4.patch, HIVE-22825.5.patch > > > With objectstores, directory lookup costs are expensive. For acid tables, it > would be good to have a directory cache to reduce number of lookup calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22863) Commit compaction txn if it is opened but compaction is skipped
[
https://issues.apache.org/jira/browse/HIVE-22863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034001#comment-17034001
]
Hive QA commented on HIVE-22863:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993067/HIVE-22863.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17992 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20545/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20545/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20545/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993067 - PreCommit-HIVE-Build
> Commit compaction txn if it is opened but compaction is skipped
> ---
>
> Key: HIVE-22863
> URL: https://issues.apache.org/jira/browse/HIVE-22863
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22863.01.patch
>
>
> Currently if a table does not have enough directories to compact, compaction
> is skipped and the compaction is either (a) marked ready for cleaning or (b)
> marked compacted. However, the txn the compaction runs in is never committed,
> it remains open, so TXNS and TXN_COMPONENTS will never be cleared of
> information about the attempted compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
[ https://issues.apache.org/jira/browse/HIVE-22815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033991#comment-17033991 ] Gopal Vijayaraghavan commented on HIVE-22815: - [~ashutoshc]: Hive is passing in the OutputFormatImpl which is a null impl, which is why the FS is not being used. > reduce the unnecessary file system object creation in MROutput > --- > > Key: HIVE-22815 > URL: https://issues.apache.org/jira/browse/HIVE-22815 > Project: Hive > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > Attachments: Hive-22815.2.patch > > > MROutput generates unnecessary file system object which may create long > latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22863) Commit compaction txn if it is opened but compaction is skipped
[
https://issues.apache.org/jira/browse/HIVE-22863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033978#comment-17033978
]
Hive QA commented on HIVE-22863:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 6 new + 37 unchanged - 0 fixed
= 43 total (was 37) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20545/dev-support/hive-personality.sh
|
| git revision | master / 46ed5c9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20545/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20545/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Commit compaction txn if it is opened but compaction is skipped
> ---
>
> Key: HIVE-22863
> URL: https://issues.apache.org/jira/browse/HIVE-22863
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22863.01.patch
>
>
> Currently if a table does not have enough directories to compact, compaction
> is skipped and the compaction is either (a) marked ready for cleaning or (b)
> marked compacted. However, the txn the compaction runs in is never committed,
> it remains open, so TXNS and TXN_COMPONENTS will never be cleared of
> information about the attempted compaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033958#comment-17033958
]
Hive QA commented on HIVE-20948:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993052/HIVE-20948.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17990 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20544/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20544/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20544/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993052 - PreCommit-HIVE-Build
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-20948.01.patch
>
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Priority: Minor (was: Major) > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Issue Type: Improvement (was: Bug) > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033943#comment-17033943 ] Miklos Gergely commented on HIVE-22864: --- [~vgarg] please review. > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Status: Patch Available (was: Open) > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Attachment: HIVE-22864.01.patch > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22864.01.patch > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22864: -- Component/s: Hive > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22864) Add option to DatabaseRule to run the Schema Tool in verbose mode for tests
[ https://issues.apache.org/jira/browse/HIVE-22864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-22864: - > Add option to DatabaseRule to run the Schema Tool in verbose mode for tests > --- > > Key: HIVE-22864 > URL: https://issues.apache.org/jira/browse/HIVE-22864 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Running the database schema tests in the metastore is always in non-verbose > mode, which can be helpful in case there is an error. Let's introduce a new > maven argument for the tests (-Dverbose.schematool) with which the output of > the Schema Tool is in verbose mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22446) Make IO decoding quantiles counters less contended resource.
[
https://issues.apache.org/jira/browse/HIVE-22446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033934#comment-17033934
]
Ashutosh Chauhan commented on HIVE-22446:
-
LGTM. [~gopalv] can you please review too?
> Make IO decoding quantiles counters less contended resource.
>
>
> Key: HIVE-22446
> URL: https://issues.apache.org/jira/browse/HIVE-22446
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22446.patch
>
>
> Currently LLAP IO relies on Hadoop Lock-based quantiles data structure and
> updates the IO decoding sample on a per batch based using.
> {code}
> org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics#addDecodeBatchTime
> {code}
> via
> {code}
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer#consumeData
> {code}
> This can be a source of thread contention.
> Goal of this ticket is to reduce the frequency of updates to avoid major
> bottleneck.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-20312: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Allow arrow clients to use their own BufferAllocator with > LlapOutputFormatService > - > > Key: HIVE-20312 > URL: https://issues.apache.org/jira/browse/HIVE-20312 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, > HIVE-20312.3.patch > > > Clients should be able to provide their own BufferAllocator to > LlapBaseInputFormat if allocator operations depend on client-side logic. For > example, clients may want to manage the allocator hierarchy per client-side > task, thread, etc.. > Currently the client is forced to use one global RootAllocator per process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033928#comment-17033928 ] Jason Dere commented on HIVE-20312: --- Thanks [~ShubhamChaurasia], I've gone ahead and committed the patch to master. > Allow arrow clients to use their own BufferAllocator with > LlapOutputFormatService > - > > Key: HIVE-20312 > URL: https://issues.apache.org/jira/browse/HIVE-20312 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, > HIVE-20312.3.patch > > > Clients should be able to provide their own BufferAllocator to > LlapBaseInputFormat if allocator operations depend on client-side logic. For > example, clients may want to manage the allocator hierarchy per client-side > task, thread, etc.. > Currently the client is forced to use one global RootAllocator per process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033926#comment-17033926
]
Hive QA commented on HIVE-20948:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 9 new + 257 unchanged - 0
fixed = 266 total (was 257) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20544/dev-support/hive-personality.sh
|
| git revision | master / 8a721db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20544/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20544/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-20948.01.patch
>
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033918#comment-17033918 ] Miklos Gergely commented on HIVE-22728: --- Fixed some issues about upgrades / init scripts, now they all work fine, verified by the method [~vgarg] suggested. > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22728: -- Attachment: (was: HIVE-22728.01.patch) > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22728: -- Attachment: HIVE-22728.01.patch > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033908#comment-17033908
]
Hive QA commented on HIVE-22765:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993048/HIVE-22765.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17990 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20543/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20543/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20543/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993048 - PreCommit-HIVE-Build
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: László Pintér
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch,
> HIVE-22765.4.patch, HIVE-22765.5.patch, HIVE-22765.6.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22856) Hive LLAP LlapArrowBatchRecordReader skipping remaining batches when ArrowStreamReader returns a 0 length batch.
[ https://issues.apache.org/jira/browse/HIVE-22856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-22856: -- Attachment: (was: HIVE-22856.01.patch) > Hive LLAP LlapArrowBatchRecordReader skipping remaining batches when > ArrowStreamReader returns a 0 length batch. > > > Key: HIVE-22856 > URL: https://issues.apache.org/jira/browse/HIVE-22856 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-22856.01.patch > > > LlapArrowBatchRecordReader returns false when the ArrowStreamReader > loadNextBatch returns column vector with 0 length. But we should keep reading > data until loadNextBatch returns false. Some batch may return column vector > of length 0, but we should ignore and wait for the next batch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22856) Hive LLAP LlapArrowBatchRecordReader skipping remaining batches when ArrowStreamReader returns a 0 length batch.
[ https://issues.apache.org/jira/browse/HIVE-22856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-22856: -- Attachment: HIVE-22856.01.patch > Hive LLAP LlapArrowBatchRecordReader skipping remaining batches when > ArrowStreamReader returns a 0 length batch. > > > Key: HIVE-22856 > URL: https://issues.apache.org/jira/browse/HIVE-22856 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-22856.01.patch > > > LlapArrowBatchRecordReader returns false when the ArrowStreamReader > loadNextBatch returns column vector with 0 length. But we should keep reading > data until loadNextBatch returns false. Some batch may return column vector > of length 0, but we should ignore and wait for the next batch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
[ https://issues.apache.org/jira/browse/HIVE-22815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033866#comment-17033866 ] Ashutosh Chauhan commented on HIVE-22815: - [~rzhappy] If MROutput doesn't use FS ever, then won't it make sense to remove that from MROutput? > reduce the unnecessary file system object creation in MROutput > --- > > Key: HIVE-22815 > URL: https://issues.apache.org/jira/browse/HIVE-22815 > Project: Hive > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > Attachments: Hive-22815.2.patch > > > MROutput generates unnecessary file system object which may create long > latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22858) HMS broken with mysql
[
https://issues.apache.org/jira/browse/HIVE-22858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033857#comment-17033857
]
Gopal Vijayaraghavan commented on HIVE-22858:
-
Never mind, I read HIVE-22804 and that was basically adding this in all the
other places.
> HMS broken with mysql
> -
>
> Key: HIVE-22858
> URL: https://issues.apache.org/jira/browse/HIVE-22858
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Using commit 7bb1d1edfcba558958265ec47245bc529eaee2d8 (Jan 27) apache master.
> Encountered the following exception when creating new database in hive (mysql
> for HMS).
> https://issues.apache.org/jira/browse/HIVE-22663 may be related to this.
>
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near '"NEXT_TXN_ID" for update' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:599)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:555)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19779)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19764)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:565)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
[jira] [Commented] (HIVE-22858) HMS broken with mysql
[
https://issues.apache.org/jira/browse/HIVE-22858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033855#comment-17033855
]
Gopal Vijayaraghavan commented on HIVE-22858:
-
That seems odd, since we should already have the ansi quotes on.
{code}
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
executeNoResult("SET @@session.sql_mode=ANSI_QUOTES");
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
statement.execute("SET @@session.sql_mode=ANSI_QUOTES");
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
stmt.execute("SET @@session.sql_mode=ANSI_QUOTES");
{code}
> HMS broken with mysql
> -
>
> Key: HIVE-22858
> URL: https://issues.apache.org/jira/browse/HIVE-22858
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Major
>
> Using commit 7bb1d1edfcba558958265ec47245bc529eaee2d8 (Jan 27) apache master.
> Encountered the following exception when creating new database in hive (mysql
> for HMS).
> https://issues.apache.org/jira/browse/HIVE-22663 may be related to this.
>
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from
> transaction database
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
> in your SQL syntax; check the manual that corresponds to your MySQL server
> version for the right syntax to use near '"NEXT_TXN_ID" for update' at line 1
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1054)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
> at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:599)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(TxnHandler.java:555)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.open_txns(HiveMetaStore.java:7956)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.open_txns(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19779)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$open_txns.getResult(ThriftHiveMetastore.java:19764)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
[jira] [Updated] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
[ https://issues.apache.org/jira/browse/HIVE-22815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang updated HIVE-22815: - Attachment: Hive-22815.2.patch > reduce the unnecessary file system object creation in MROutput > --- > > Key: HIVE-22815 > URL: https://issues.apache.org/jira/browse/HIVE-22815 > Project: Hive > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > Attachments: Hive-22815.2.patch > > > MROutput generates unnecessary file system object which may create long > latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
[ https://issues.apache.org/jira/browse/HIVE-22815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang updated HIVE-22815: - Attachment: (was: HIVE-222815.3.patch) > reduce the unnecessary file system object creation in MROutput > --- > > Key: HIVE-22815 > URL: https://issues.apache.org/jira/browse/HIVE-22815 > Project: Hive > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > > MROutput generates unnecessary file system object which may create long > latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
[ https://issues.apache.org/jira/browse/HIVE-22815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang updated HIVE-22815: - Attachment: (was: Hive-222815.2.patch) > reduce the unnecessary file system object creation in MROutput > --- > > Key: HIVE-22815 > URL: https://issues.apache.org/jira/browse/HIVE-22815 > Project: Hive > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > > MROutput generates unnecessary file system object which may create long > latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
[ https://issues.apache.org/jira/browse/HIVE-22815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang updated HIVE-22815: - Attachment: (was: Hive-222815.1.patch) > reduce the unnecessary file system object creation in MROutput > --- > > Key: HIVE-22815 > URL: https://issues.apache.org/jira/browse/HIVE-22815 > Project: Hive > Issue Type: Bug >Reporter: Richard Zhang >Assignee: Richard Zhang >Priority: Major > > MROutput generates unnecessary file system object which may create long > latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033853#comment-17033853
]
Hive QA commented on HIVE-22765:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 22 new + 55 unchanged - 21 fixed = 77 total (was 76) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20543/dev-support/hive-personality.sh
|
| git revision | master / 8a721db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20543/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20543/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: László Pintér
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch,
> HIVE-22765.4.patch, HIVE-22765.5.patch, HIVE-22765.6.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
>
[jira] [Commented] (HIVE-22610) Minor compaction for MM (insert-only) tables
[
https://issues.apache.org/jira/browse/HIVE-22610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033838#comment-17033838
]
Hive QA commented on HIVE-22610:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993045/HIVE-22610.02.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18002 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20542/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20542/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20542/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993045 - PreCommit-HIVE-Build
> Minor compaction for MM (insert-only) tables
>
>
> Key: HIVE-22610
> URL: https://issues.apache.org/jira/browse/HIVE-22610
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22610.01.patch, HIVE-22610.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[
https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-22762:
-
Attachment: HIVE-22762.01.patch
Status: Patch Available (was: Open)
> Leap day is incorrectly parsed during cast in Hive
> --
>
> Key: HIVE-22762
> URL: https://issues.apache.org/jira/browse/HIVE-22762
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22762.01.patch, HIVE-22762.01.patch,
> HIVE-22762.01.patch, HIVE-22762.01.patch
>
>
> While casting a string to a date with a custom date format having day token
> before year and moth tokens, the date is parsed incorrectly for leap days.
> h3. How to reproduce
> Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with
> Hive. The query results in *2020-02-28*, incorrectly.
>
> Executing the another cast with a slightly modified representation of the
> date (day is preceded by year and moth) is however correctly parsed:
> {code}select cast("0 02 29" as date format "rr mm dd"){code}
> It returns *2020-02-29*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22763) 0 is accepted in 12-hour format during timestamp cast
[
https://issues.apache.org/jira/browse/HIVE-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-22763:
-
Attachment: HIVE-22763.01.patch
Status: Patch Available (was: Open)
> 0 is accepted in 12-hour format during timestamp cast
> -
>
> Key: HIVE-22763
> URL: https://issues.apache.org/jira/browse/HIVE-22763
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22763.01.patch, HIVE-22763.01.patch,
> HIVE-22763.01.patch, HIVE-22763.01.patch
>
>
> Having a timestamp string in 12-hour format can be parsed if the hour is 0,
> however, based on the [design
> document|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit],
> it should be rejected.
> h3. How to reproduce
> Run {code}select cast("2020-01-01 0 am 00" as timestamp format "-mm-dd
> hh12 p.m. ss"){code}
> It shouldn' t be parsed, as the hour component is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-10362) Support Type check/conversion in dynamic partition column
[ https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-10362: - Attachment: HIVE-10362.12.patch Status: Patch Available (was: Open) > Support Type check/conversion in dynamic partition column > - > > Key: HIVE-10362 > URL: https://issues.apache.org/jira/browse/HIVE-10362 > Project: Hive > Issue Type: Improvement > Components: Query Processor, Types >Affects Versions: 3.0.0, 1.0.0, 4.0.0 >Reporter: Chaoyu Tang >Assignee: Karen Coppage >Priority: Major > Labels: breaking_change > Fix For: 4.0.0 > > Attachments: HIVE-10362.01.patch, HIVE-10362.02.patch, > HIVE-10362.03.patch, HIVE-10362.04.patch, HIVE-10362.04.patch, > HIVE-10362.06.patch, HIVE-10362.07.patch, HIVE-10362.07.patch, > HIVE-10362.08.patch, HIVE-10362.09.patch, HIVE-10362.10.patch, > HIVE-10362.11.patch, HIVE-10362.12.patch, HIVE-10362.12.patch, > HIVE-10362.12.patch, HIVE-10362.12.patch, HIVE-10362.12.patch > > > There are quite a lot of issues associated with the non-noramlized or > type-mismatched values for partition column. Hive has many ways to introduce > such problematic data. HIVE-10307 mainly provides the support to type > check/convert/normalize the partition column value in static partition > specification. This JIRA tries to deal with the partition column type in > dynamic partition insert. Currently any data can be inserted as a partition > column value as long as it is quoted as a string. For example, > create table dynparttypechecknum (key int, value string) partitioned by (part > int); > insert into dynparttypechecknum partition (part) select key, value, '1' > from src limit 1; > show partitions dynparttypechecknum; > -- > part=1 > The partition column value is non-normalized int 1. It causes some > unnecessary problems such as integer partition column JDO filter pushdown > (see HIVE-6052) and others like HIVE-10210. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22763) 0 is accepted in 12-hour format during timestamp cast
[
https://issues.apache.org/jira/browse/HIVE-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-22763:
-
Status: Open (was: Patch Available)
> 0 is accepted in 12-hour format during timestamp cast
> -
>
> Key: HIVE-22763
> URL: https://issues.apache.org/jira/browse/HIVE-22763
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22763.01.patch, HIVE-22763.01.patch,
> HIVE-22763.01.patch, HIVE-22763.01.patch
>
>
> Having a timestamp string in 12-hour format can be parsed if the hour is 0,
> however, based on the [design
> document|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit],
> it should be rejected.
> h3. How to reproduce
> Run {code}select cast("2020-01-01 0 am 00" as timestamp format "-mm-dd
> hh12 p.m. ss"){code}
> It shouldn' t be parsed, as the hour component is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[
https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-22762:
-
Status: Open (was: Patch Available)
> Leap day is incorrectly parsed during cast in Hive
> --
>
> Key: HIVE-22762
> URL: https://issues.apache.org/jira/browse/HIVE-22762
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22762.01.patch, HIVE-22762.01.patch,
> HIVE-22762.01.patch
>
>
> While casting a string to a date with a custom date format having day token
> before year and moth tokens, the date is parsed incorrectly for leap days.
> h3. How to reproduce
> Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with
> Hive. The query results in *2020-02-28*, incorrectly.
>
> Executing the another cast with a slightly modified representation of the
> date (day is preceded by year and moth) is however correctly parsed:
> {code}select cast("0 02 29" as date format "rr mm dd"){code}
> It returns *2020-02-29*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22610) Minor compaction for MM (insert-only) tables
[
https://issues.apache.org/jira/browse/HIVE-22610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033795#comment-17033795
]
Hive QA commented on HIVE-22610:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 1 new + 606 unchanged - 0
fixed = 607 total (was 606) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 8
unchanged - 3 fixed = 10 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20542/dev-support/hive-personality.sh
|
| git revision | master / 8a721db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20542/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20542/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20542/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Minor compaction for MM (insert-only) tables
>
>
> Key: HIVE-22610
> URL: https://issues.apache.org/jira/browse/HIVE-22610
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22610.01.patch, HIVE-22610.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033796#comment-17033796 ] Miklos Gergely commented on HIVE-22728: --- [~vgarg] thanks! is there also a way to test derby upgrade? > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Status: Patch Available (was: Open)
Reattaching patch since no test run on the patch.
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Status: Open (was: Patch Available)
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Attachment: HIVE-22841.1.patch
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Attachment: (was: HIVE-22841-1.patch)
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033768#comment-17033768 ] Vineet Garg commented on HIVE-22728: [~mgergely] FYI we have metastore docker tests which test installation and upgrade against all four dbs. You can find more info in standalone-metastore/DEV-README > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22781) Add ability to immediately execute a scheduled query
[
https://issues.apache.org/jira/browse/HIVE-22781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033760#comment-17033760
]
Hive QA commented on HIVE-22781:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993023/HIVE-22781.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17992 tests
executed
*Failed tests:*
{noformat}
TestNumMetastoreCallsObjectStore - did not produce a TEST-*.xml file (likely
timed out) (batchId=359)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
(batchId=199)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20540/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20540/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20540/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993023 - PreCommit-HIVE-Build
> Add ability to immediately execute a scheduled query
>
>
> Key: HIVE-22781
> URL: https://issues.apache.org/jira/browse/HIVE-22781
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22781.01.patch, HIVE-22781.02.patch,
> HIVE-22781.03.patch, HIVE-22781.04.patch, HIVE-22781.04.patch,
> HIVE-22781.04.patch, HIVE-22781.05.patch
>
>
> there are some differences when the system invokes the scheduled query / the
> user executes it in a shell - forcing the schedule to run might be usefull in
> developing/debugging schedules
> something like:
> {code}
> alter scheduled query a execute
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20312) Allow arrow clients to use their own BufferAllocator with LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033757#comment-17033757 ] Shubham Chaurasia commented on HIVE-20312: -- [~jdere] [~maheshk114] Looks like this was not merged. Can you please have a look and merge ? cc [~anishek] [~thejas] > Allow arrow clients to use their own BufferAllocator with > LlapOutputFormatService > - > > Key: HIVE-20312 > URL: https://issues.apache.org/jira/browse/HIVE-20312 > Project: Hive > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-20312.1.patch, HIVE-20312.2.patch, > HIVE-20312.3.patch > > > Clients should be able to provide their own BufferAllocator to > LlapBaseInputFormat if allocator operations depend on client-side logic. For > example, clients may want to manage the allocator hierarchy per client-side > task, thread, etc.. > Currently the client is forced to use one global RootAllocator per process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-22783) Add test for HIVE-22366
[ https://issues.apache.org/jira/browse/HIVE-22783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reopened HIVE-22783: Reverted from master for now since this test was timing out > Add test for HIVE-22366 > --- > > Key: HIVE-22783 > URL: https://issues.apache.org/jira/browse/HIVE-22783 > Project: Hive > Issue Type: Sub-task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22783.1.patch, HIVE-22783.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22856) Hive LLAP LlapArrowBatchRecordReader skipping remaining batches when ArrowStreamReader returns a 0 length batch.
[ https://issues.apache.org/jira/browse/HIVE-22856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033745#comment-17033745 ] Jason Dere commented on HIVE-22856: --- +1 nice catch > Hive LLAP LlapArrowBatchRecordReader skipping remaining batches when > ArrowStreamReader returns a 0 length batch. > > > Key: HIVE-22856 > URL: https://issues.apache.org/jira/browse/HIVE-22856 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-22856.01.patch > > > LlapArrowBatchRecordReader returns false when the ArrowStreamReader > loadNextBatch returns column vector with 0 length. But we should keep reading > data until loadNextBatch returns false. Some batch may return column vector > of length 0, but we should ignore and wait for the next batch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033732#comment-17033732 ] Zoltan Haindrich commented on HIVE-21000: - I think this was stopped to wait for the accumulo 2 release ( since they were already putting out alphas or something) [~q.xu]: I don't know about [~isuller]'s availability; but the plan was to upgrade accumulo and then continue with Thriftsome time ago I also bumped into some performance issues with 0.9.x ; so upgrading would be the best > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22781) Add ability to immediately execute a scheduled query
[
https://issues.apache.org/jira/browse/HIVE-22781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033722#comment-17033722
]
Hive QA commented on HIVE-22781:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
54s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 12 unchanged - 0 fixed
= 14 total (was 12) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20540/dev-support/hive-personality.sh
|
| git revision | master / 0d9deba |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20540/yetus/diff-checkstyle-ql.txt
|
| modules | C: parser standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20540/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add ability to immediately execute a scheduled query
>
>
> Key: HIVE-22781
> URL: https://issues.apache.org/jira/browse/HIVE-22781
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22781.01.patch, HIVE-22781.02.patch,
> HIVE-22781.03.patch, HIVE-22781.04.patch, HIVE-22781.04.patch,
> HIVE-22781.04.patch, HIVE-22781.05.patch
>
>
> there are some differences when the system invokes the scheduled query / the
> user executes it in a shell - forcing the schedule to run might be usefull in
> developing/debugging schedules
> something like:
> {code}
> alter scheduled query a execute
> {code}
--
This message was sent by Atlassian Jira
[jira] [Commented] (HIVE-14165) Remove Hive file listing during split computation
[ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033713#comment-17033713 ] Sahil Takiar commented on HIVE-14165: - Marking as unassigned as I am no longer working on this. IIRC this speedup only applies to very simply queries - e.g. select / project queries. > Remove Hive file listing during split computation > - > > Key: HIVE-14165 > URL: https://issues.apache.org/jira/browse/HIVE-14165 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Abdullah Yousufi >Priority: Major > Attachments: HIVE-14165.02.patch, HIVE-14165.03.patch, > HIVE-14165.04.patch, HIVE-14165.05.patch, HIVE-14165.06.patch, > HIVE-14165.07.patch, HIVE-14165.patch > > > The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's > FileInputFormat.java will list the files during split computation anyway to > determine their size. One way to remove this is to catch the > InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the > Hive side instead of doing the file listing beforehand. > For S3 select queries on partitioned tables, this results in a 2x speedup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-14165) Remove Hive file listing during split computation
[ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-14165: --- Assignee: (was: Sahil Takiar) > Remove Hive file listing during split computation > - > > Key: HIVE-14165 > URL: https://issues.apache.org/jira/browse/HIVE-14165 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Abdullah Yousufi >Priority: Major > Attachments: HIVE-14165.02.patch, HIVE-14165.03.patch, > HIVE-14165.04.patch, HIVE-14165.05.patch, HIVE-14165.06.patch, > HIVE-14165.07.patch, HIVE-14165.patch > > > The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's > FileInputFormat.java will list the files during split computation anyway to > determine their size. One way to remove this is to catch the > InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the > Hive side instead of doing the file listing beforehand. > For S3 select queries on partitioned tables, this results in a 2x speedup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
[
https://issues.apache.org/jira/browse/HIVE-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramesh Kumar Thangarajan updated HIVE-22828:
Status: Open (was: Patch Available)
> Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
> -
>
> Key: HIVE-22828
> URL: https://issues.apache.org/jira/browse/HIVE-22828
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal Vijayaraghavan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22828.1.patch, HIVE-22828.3.patch
>
>
> {code}
> select
> sum(case when (ss_item_sk=1) then ss_sales_price else null end),
> sum(case when (ss_item_sk=2) then ss_sales_price else ss_sales_price+1 end),
> sum(case when (ss_item_sk=2) then 1.0BD+ss_sales_price else null end)
> from store_sales where ss_sold_date_sk % = 1;
> {code}
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal.aggregateInputSelection(VectorUDAFSumDecimal.java:113)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:221)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java#L3950
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
[
https://issues.apache.org/jira/browse/HIVE-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramesh Kumar Thangarajan updated HIVE-22828:
Attachment: HIVE-22828.3.patch
Status: Patch Available (was: Open)
> Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
> -
>
> Key: HIVE-22828
> URL: https://issues.apache.org/jira/browse/HIVE-22828
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal Vijayaraghavan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22828.1.patch, HIVE-22828.3.patch
>
>
> {code}
> select
> sum(case when (ss_item_sk=1) then ss_sales_price else null end),
> sum(case when (ss_item_sk=2) then ss_sales_price else ss_sales_price+1 end),
> sum(case when (ss_item_sk=2) then 1.0BD+ss_sales_price else null end)
> from store_sales where ss_sold_date_sk % = 1;
> {code}
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal.aggregateInputSelection(VectorUDAFSumDecimal.java:113)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:221)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java#L3950
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
[
https://issues.apache.org/jira/browse/HIVE-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramesh Kumar Thangarajan updated HIVE-22828:
Attachment: (was: HIVE-22828.3.patch)
> Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
> -
>
> Key: HIVE-22828
> URL: https://issues.apache.org/jira/browse/HIVE-22828
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal Vijayaraghavan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22828.1.patch, HIVE-22828.3.patch
>
>
> {code}
> select
> sum(case when (ss_item_sk=1) then ss_sales_price else null end),
> sum(case when (ss_item_sk=2) then ss_sales_price else ss_sales_price+1 end),
> sum(case when (ss_item_sk=2) then 1.0BD+ss_sales_price else null end)
> from store_sales where ss_sold_date_sk % = 1;
> {code}
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal.aggregateInputSelection(VectorUDAFSumDecimal.java:113)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:221)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java#L3950
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22828) Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
[
https://issues.apache.org/jira/browse/HIVE-22828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramesh Kumar Thangarajan updated HIVE-22828:
Attachment: (was: HIVE-22828.3.patch)
> Decimal64: NVL & CASE statements implicitly convert decimal64 to 128
> -
>
> Key: HIVE-22828
> URL: https://issues.apache.org/jira/browse/HIVE-22828
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Gopal Vijayaraghavan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22828.1.patch, HIVE-22828.3.patch
>
>
> {code}
> select
> sum(case when (ss_item_sk=1) then ss_sales_price else null end),
> sum(case when (ss_item_sk=2) then ss_sales_price else ss_sales_price+1 end),
> sum(case when (ss_item_sk=2) then 1.0BD+ss_sales_price else null end)
> from store_sales where ss_sold_date_sk % = 1;
> {code}
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.Decimal64ColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:125)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.VectorUDAFSumDecimal.aggregateInputSelection(VectorUDAFSumDecimal.java:113)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:221)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:414)
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java#L3950
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[
https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033688#comment-17033688
]
Hive QA commented on HIVE-22762:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993013/HIVE-22762.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17991 tests
executed
*Failed tests:*
{noformat}
TestNumMetastoreCallsObjectStore - did not produce a TEST-*.xml file (likely
timed out) (batchId=359)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentLineage (batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20539/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20539/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20539/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993013 - PreCommit-HIVE-Build
> Leap day is incorrectly parsed during cast in Hive
> --
>
> Key: HIVE-22762
> URL: https://issues.apache.org/jira/browse/HIVE-22762
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22762.01.patch, HIVE-22762.01.patch,
> HIVE-22762.01.patch
>
>
> While casting a string to a date with a custom date format having day token
> before year and moth tokens, the date is parsed incorrectly for leap days.
> h3. How to reproduce
> Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with
> Hive. The query results in *2020-02-28*, incorrectly.
>
> Executing the another cast with a slightly modified representation of the
> date (day is preceded by year and moth) is however correctly parsed:
> {code}select cast("0 02 29" as date format "rr mm dd"){code}
> It returns *2020-02-29*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22863) Commit compaction txn if it is opened but compaction is skipped
[ https://issues.apache.org/jira/browse/HIVE-22863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22863: - Attachment: HIVE-22863.01.patch Status: Patch Available (was: Open) > Commit compaction txn if it is opened but compaction is skipped > --- > > Key: HIVE-22863 > URL: https://issues.apache.org/jira/browse/HIVE-22863 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-22863.01.patch > > > Currently if a table does not have enough directories to compact, compaction > is skipped and the compaction is either (a) marked ready for cleaning or (b) > marked compacted. However, the txn the compaction runs in is never committed, > it remains open, so TXNS and TXN_COMPONENTS will never be cleared of > information about the attempted compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22863) Commit compaction txn if it is opened but compaction is skipped
[ https://issues.apache.org/jira/browse/HIVE-22863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-22863: > Commit compaction txn if it is opened but compaction is skipped > --- > > Key: HIVE-22863 > URL: https://issues.apache.org/jira/browse/HIVE-22863 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > Currently if a table does not have enough directories to compact, compaction > is skipped and the compaction is either (a) marked ready for cleaning or (b) > marked compacted. However, the txn the compaction runs in is never committed, > it remains open, so TXNS and TXN_COMPONENTS will never be cleared of > information about the attempted compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[
https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033666#comment-17033666
]
Miklos Gergely commented on HIVE-22728:
---
Ok, I've uploaded the patch, while I'm still trying to test it against
different db types, and also trying to test the upgrade.
I've modified the error message as [~kgyrtkirk] suggested though I'm not sure
if catalog should be included in it for now, as it would always look like this:
{code:java}
hive..{code}
So as long as there is only one default catalog, I'd suggest not to include
it's name in the error message. Please let me know what you think.
> Limit the scope of uniqueness of constraint name to database
>
>
> Key: HIVE-22728
> URL: https://issues.apache.org/jira/browse/HIVE-22728
> Project: Hive
> Issue Type: Wish
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-22728.01.patch
>
>
> Currently, constraint names are globally unique across all databases
> (assumption is that this may have done by design). Nevertheless, though
> behavior seems to be implementation specific, it would be interesting to
> limit the scope to uniqueness per database.
> Currently we do not store database information with the constraints. To
> change the scope to one db, we would need to store the DB_ID in the
> KEY_CONSTRAINTS table in metastore when we create a constraint and add the
> DB_ID to the PRIMARY KEY of that table. Some minor changes to the error
> messages would be needed too, since otherwise it would be difficult to
> identify the correct violation in queries that span across multiple
> databases. Additionally, the SQL scripts will need to be updated to populate
> the DB_ID when we upgrade to new version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22862) Remove unnecessary calls to isEnoughToCompact
[ https://issues.apache.org/jira/browse/HIVE-22862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22862: - Attachment: HIVE-22862.01.patch Status: Patch Available (was: Open) > Remove unnecessary calls to isEnoughToCompact > - > > Key: HIVE-22862 > URL: https://issues.apache.org/jira/browse/HIVE-22862 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Attachments: HIVE-22862.01.patch > > > QueryCompactor.Util#isEnoughToCompact is called in Worker#run once before any > sort of compaction is run, after this it is called in 3 other places during > compaction unnecessarily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22728: -- Attachment: HIVE-22728.01.patch > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22728) Limit the scope of uniqueness of constraint name to database
[ https://issues.apache.org/jira/browse/HIVE-22728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22728: -- Status: Patch Available (was: Open) > Limit the scope of uniqueness of constraint name to database > > > Key: HIVE-22728 > URL: https://issues.apache.org/jira/browse/HIVE-22728 > Project: Hive > Issue Type: Wish >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-22728.01.patch > > > Currently, constraint names are globally unique across all databases > (assumption is that this may have done by design). Nevertheless, though > behavior seems to be implementation specific, it would be interesting to > limit the scope to uniqueness per database. > Currently we do not store database information with the constraints. To > change the scope to one db, we would need to store the DB_ID in the > KEY_CONSTRAINTS table in metastore when we create a constraint and add the > DB_ID to the PRIMARY KEY of that table. Some minor changes to the error > messages would be needed too, since otherwise it would be difficult to > identify the correct violation in queries that span across multiple > databases. Additionally, the SQL scripts will need to be updated to populate > the DB_ID when we upgrade to new version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22862) Remove unnecessary calls to isEnoughToCompact
[ https://issues.apache.org/jira/browse/HIVE-22862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-22862: > Remove unnecessary calls to isEnoughToCompact > - > > Key: HIVE-22862 > URL: https://issues.apache.org/jira/browse/HIVE-22862 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > > QueryCompactor.Util#isEnoughToCompact is called in Worker#run once before any > sort of compaction is run, after this it is called in 3 other places during > compaction unnecessarily. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Pintér updated HIVE-20948:
-
Attachment: HIVE-20948.01.patch
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-20948.01.patch
>
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Pintér updated HIVE-20948:
-
Status: Patch Available (was: Open)
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-20948.01.patch
>
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Pintér reassigned HIVE-20948:
Assignee: László Pintér
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: László Pintér
>Priority: Major
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22762) Leap day is incorrectly parsed during cast in Hive
[
https://issues.apache.org/jira/browse/HIVE-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033636#comment-17033636
]
Hive QA commented on HIVE-22762:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} common generated 1 new + 63 unchanged - 0 fixed = 64
total (was 63) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| |
org.apache.hadoop.hive.common.format.datetime.HiveSqlDateTimeFormatter$Token
defines compareTo(Object) and uses Object.equals() At
HiveSqlDateTimeFormatter.java:Object.equals() At
HiveSqlDateTimeFormatter.java:[lines 572-578] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20539/dev-support/hive-personality.sh
|
| git revision | master / 0d9deba |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20539/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20539/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Leap day is incorrectly parsed during cast in Hive
> --
>
> Key: HIVE-22762
> URL: https://issues.apache.org/jira/browse/HIVE-22762
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22762.01.patch, HIVE-22762.01.patch,
> HIVE-22762.01.patch
>
>
> While casting a string to a date with a custom date format having day token
> before year and moth tokens, the date is parsed incorrectly for leap days.
> h3. How to reproduce
> Execute {code}select cast("29 02 0" as date format "dd mm rr"){code} with
> Hive. The query results in *2020-02-28*, incorrectly.
>
> Executing the another cast with a slightly modified representation of the
> date (day is preceded by year and moth) is however correctly parsed:
> {code}select cast("0 02 29" as date format "rr mm dd"){code}
> It returns *2020-02-29*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22763) 0 is accepted in 12-hour format during timestamp cast
[
https://issues.apache.org/jira/browse/HIVE-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033625#comment-17033625
]
Hive QA commented on HIVE-22763:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12993012/HIVE-22763.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17991 tests
executed
*Failed tests:*
{noformat}
TestNumMetastoreCallsObjectStore - did not produce a TEST-*.xml file (likely
timed out) (batchId=359)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20538/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20538/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20538/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12993012 - PreCommit-HIVE-Build
> 0 is accepted in 12-hour format during timestamp cast
> -
>
> Key: HIVE-22763
> URL: https://issues.apache.org/jira/browse/HIVE-22763
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22763.01.patch, HIVE-22763.01.patch,
> HIVE-22763.01.patch
>
>
> Having a timestamp string in 12-hour format can be parsed if the hour is 0,
> however, based on the [design
> document|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit],
> it should be rejected.
> h3. How to reproduce
> Run {code}select cast("2020-01-01 0 am 00" as timestamp format "-mm-dd
> hh12 p.m. ss"){code}
> It shouldn' t be parsed, as the hour component is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-14165) Remove Hive file listing during split computation
[ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17033607#comment-17033607 ] Steve Loughran commented on HIVE-14165: --- What is the current status of this? Is it a defacto WONTFIX? Or is someone keeping the patch up to date > Remove Hive file listing during split computation > - > > Key: HIVE-14165 > URL: https://issues.apache.org/jira/browse/HIVE-14165 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Abdullah Yousufi >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-14165.02.patch, HIVE-14165.03.patch, > HIVE-14165.04.patch, HIVE-14165.05.patch, HIVE-14165.06.patch, > HIVE-14165.07.patch, HIVE-14165.patch > > > The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's > FileInputFormat.java will list the files during split computation anyway to > determine their size. One way to remove this is to catch the > InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the > Hive side instead of doing the file listing beforehand. > For S3 select queries on partitioned tables, this results in a 2x speedup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Chovan updated HIVE-22765:
-
Attachment: HIVE-22765.6.patch
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: László Pintér
>Assignee: Zoltan Chovan
>Priority: Major
> Attachments: HIVE-22765.2.patch, HIVE-22765.3.patch,
> HIVE-22765.4.patch, HIVE-22765.5.patch, HIVE-22765.6.patch, HIVE-22765.patch
>
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
