[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Status: Open (was: Patch Available) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638438#comment-16638438 ] Ashutosh Bapat edited comment on HIVE-20644 at 10/5/18 5:25 AM: Attaching patch with [~thejas]as's comments addressed. I ran the three testcases which failed with the previous patch separately. All of those passed on my machine. Also I observe that the diffs are unrelated to the patch. was (Author: ashutosh.bapat): Attaching patch with Thejas's comments addressed. I ran the three testcases which failed with the previous patch separately. All of those passed on my machine. Also I observe that the diffs are unrelated to the patch. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03, > HIVE-20644.04, HIVE-20644.05 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639288#comment-16639288
]
Hive QA commented on HIVE-20610:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942405/HIVE-20610.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15025 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14245/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14245/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14245/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942405 - PreCommit-HIVE-Build
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
> 2018-09-19T06:42:05,072 ERROR [main] utils.MetaStoreUtils: Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException Unable to delete
> directory: file:/tmp
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to delete
> directory: file:/tmp
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreFsImpl.deleteDir(HiveMetaStoreFsImpl.java:45)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:365)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Warehouse.deleteDir(Warehouse.java:353)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.deleteTableData(HiveMetaStore.java:2562)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:2523)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:2685)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_102]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_102]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Attachment: HIVE-20678.2.patch Status: Patch Available (was: Open) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Attachment: (was: HIVE-20678.2.patch) > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20651) JdbcStorageHandler password should be encrypted
[
https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20651:
--
Attachment: HIVE-20651.4.patch
> JdbcStorageHandler password should be encrypted
> ---
>
> Key: HIVE-20651
> URL: https://issues.apache.org/jira/browse/HIVE-20651
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch,
> HIVE-20651.3.patch, HIVE-20651.4.patch
>
>
> Currently, external jdbc table with JdbcStorageHandler store password as
> "hive.sql.dbcp.password" table property in clear text. We should put it in a
> keystore file. Here is the proposed change:
> {code:java}
> ….
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.dbcp.password.keystore" =
> "hdfs:///user/hive/credential/postgres.jceks",
> "hive.sql.dbcp.password.key" = "mydb.password"
> );
> {code}
>
> The jceks file is created with:
> {code}
> hadoop credential create mydb.password -provider
> hdfs:///user/hive/credential/postgres.jceks -v secretpassword
> {code}
> User can choose to put all db password in one jceks, or a separate jceks for
> each db.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639269#comment-16639269 ] Alice Fan commented on HIVE-20678: -- Hi [~ychena] [~ngangam], Could you please help to review this? Thanks! > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20494) GenericUDFRestrictInformationSchema is broken after HIVE-19440
[ https://issues.apache.org/jira/browse/HIVE-20494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20494: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 4.0.0 Status: Resolved (was: Patch Available) > GenericUDFRestrictInformationSchema is broken after HIVE-19440 > -- > > Key: HIVE-20494 > URL: https://issues.apache.org/jira/browse/HIVE-20494 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20494.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Fan updated HIVE-20678: - Attachment: HIVE-20678.2.patch > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[ https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639258#comment-16639258 ] Alice Fan commented on HIVE-20678: -- Submit patch 2 to improve style errors. > HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure > compatibility > > > Key: HIVE-20678 > URL: https://issues.apache.org/jira/browse/HIVE-20678 > Project: Hive > Issue Type: Bug > Components: HBase Handler, Hive, Spark >Affects Versions: 4.0.0 >Reporter: Alice Fan >Assignee: Alice Fan >Priority: Major > Attachments: HIVE-20678.1.patch, HIVE-20678.2.patch > > > This issue was caused by same reason described in > https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems > to be abandoned, so create this one to resolve the issue. > HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will > break the compatibility in spark's API when it is using hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20610) TestDbNotificationListener should not use /tmp directory
[
https://issues.apache.org/jira/browse/HIVE-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639247#comment-16639247
]
Hive QA commented on HIVE-20610:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14245/dev-support/hive-personality.sh
|
| git revision | master / f0434c5 |
| Default Java | 1.8.0_111 |
| modules | C: itests/hcatalog-unit U: itests/hcatalog-unit |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14245/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestDbNotificationListener should not use /tmp directory
>
>
> Key: HIVE-20610
> URL: https://issues.apache.org/jira/browse/HIVE-20610
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 4.0.0
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20610.1.patch, HIVE-20610.2.patch,
> HIVE-20610.3.patch, HIVE-20610.4.patch
>
>
> Using /tmp directory creates exceptions for tests like dropTable :
> {code:java}
> 2018-09-19T06:42:04,818 INFO [main] metastore.HiveMetaStore: 0: drop_table :
> tbl=hive.default.droptbl
> 2018-09-19T06:42:04,819 INFO [main] HiveMetaStore.audit: ugi=hiveptest
> ip=unknown-ip-addr cmd=drop_table : tbl=hive.default.droptbl
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.ICE-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.XIM-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.X11-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/hsperfdata_root]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.font-unix]: it still exists.
> 2018-09-19T06:42:05,072 WARN [main] fs.FileUtil: Failed to delete file or
> dir [/tmp/.Test-unix]: it still exists.
> 2018-09-19T06:42:05,072 ERROR [main] utils.FileUtils: Failed to delete
> file:/tmp
>
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Attachment: HIVE-20644.05 Status: Patch Available (was: In Progress) Addresses whitespace and checkstyle complaints by Hive QA. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03, > HIVE-20644.04, HIVE-20644.05 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Status: In Progress (was: Patch Available) > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03, > HIVE-20644.04 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Attachment: HIVE-20646.03.patch > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch, > HIVE-20646.03.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Status: Open (was: Patch Available) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch, > HIVE-20646.03.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Status: Patch Available (was: Open) Attached 03.patch after rebasing with master > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch, > HIVE-20646.03.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639236#comment-16639236
]
Hive QA commented on HIVE-20644:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942409/HIVE-20644.04
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15025 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14244/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14244/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14244/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942409 - PreCommit-HIVE-Build
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03,
> HIVE-20644.04
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
> This way some sensitive information is leaked to a user through exception
> message. That information may not be available to the user otherwise. Hence
> it's a kind of security breach or violation of access control.
> The contents of the row or the arguments to a function may be useful for
> debugging and hence it's worth to add those to logs. Hence proposal here to
> log a separate message with log level DEBUG or INFO containing the string
> representation of the row. Users can configure their logging so that
> DEBUG/INFO messages do not go to the client but at the same time are
> available in the hive server logs for debugging. The actual exception message
> will not contain any sensitive data like row data or argument data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18785) Make JSON SerDe First-Class SerDe
[
https://issues.apache.org/jira/browse/HIVE-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639224#comment-16639224
]
BELUGA BEHR commented on HIVE-18785:
Ya [~aihuaxu]. Thank you for doing [HIVE-19899]. However, I put this ticket
together less as a single task and more just a place to aggregate and connect
all the JSON related tickets floating around out there.
> Make JSON SerDe First-Class SerDe
> -
>
> Key: HIVE-18785
> URL: https://issues.apache.org/jira/browse/HIVE-18785
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Aihua Xu
>Priority: Major
>
> According to the [Hive SerDe
> Docs|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats],
> there are some extra steps involved in getting the JSON SerDe to work:
> {quote}
> ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> In some distributions, a reference to hive-hcatalog-core.jar is required.
> ADD JAR /usr/lib/hive-hcatalog/lib/hive-hcatalog-core.jar;
> {quote}
> I would like to propose that we move this SerDe into first-class status:
> {{STORED AS JSONFILE}}
> The user should have to perform no additional steps to use this SerDe.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18785) Make JSON SerDe First-Class SerDe
[
https://issues.apache.org/jira/browse/HIVE-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639224#comment-16639224
]
BELUGA BEHR edited comment on HIVE-18785 at 10/5/18 3:44 AM:
-
Ya [~aihuaxu]. Thank you for doing [HIVE-19899]. However, I put this ticket
together less as a single task and more just a place to aggregate and connect
all the JSON related tickets floating around out there. We can close this one
once all the outstanding issues are addressed.
was (Author: belugabehr):
Ya [~aihuaxu]. Thank you for doing [HIVE-19899]. However, I put this ticket
together less as a single task and more just a place to aggregate and connect
all the JSON related tickets floating around out there.
> Make JSON SerDe First-Class SerDe
> -
>
> Key: HIVE-18785
> URL: https://issues.apache.org/jira/browse/HIVE-18785
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Aihua Xu
>Priority: Major
>
> According to the [Hive SerDe
> Docs|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats],
> there are some extra steps involved in getting the JSON SerDe to work:
> {quote}
> ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> In some distributions, a reference to hive-hcatalog-core.jar is required.
> ADD JAR /usr/lib/hive-hcatalog/lib/hive-hcatalog-core.jar;
> {quote}
> I would like to propose that we move this SerDe into first-class status:
> {{STORED AS JSONFILE}}
> The user should have to perform no additional steps to use this SerDe.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[
https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639222#comment-16639222
]
Hive QA commented on HIVE-20644:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 9 new + 161 unchanged - 0
fixed = 170 total (was 161) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 12 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14244/dev-support/hive-personality.sh
|
| git revision | master / f0434c5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14244/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14244/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14244/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Avoid exposing sensitive infomation through a Hive Runtime exception
>
>
> Key: HIVE-20644
> URL: https://issues.apache.org/jira/browse/HIVE-20644
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03,
> HIVE-20644.04
>
>
> The HiveException raised from the following methods is exposing the datarow
> the caused the run time exception.
> # ReduceRecordSource::GroupIterator::next() - around line 372
> # MapOperator::process() - around line 567
> # ExecReducer::reduce() - around line 243
> In all the cases, a string representation of the row is constructed on the
> fly and is included in
> the error message.
> VectorMapOperator::process() - around line 973 raises the same exception but
> it's not exposing the row since the row contents are not included in the
> error message.
> While trying to reproduce above error, I also found that the arguments to a
> UDF get exposed in log messages from FunctionRegistry::invoke() around line
> 1114. This too can cause sensitive information to be leaked through error
> message.
> This way some
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639220#comment-16639220 ] Peter Vary commented on HIVE-20647: --- +1 > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch, > HIVE-20647.6.patch, HIVE-20647.7.patch, HIVE-20647.8.patch, HIVE-20647.9.patch > > > * Removed obsolete hadoopVer from QTestUtil > * Cleaned up QTestUtil, QTestArgumentsBuilder > * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation: > (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR: Uninitialized read of field > method called from constructor of superclass) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20590) Allow merge statement to have column schema
[
https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639208#comment-16639208
]
Hive QA commented on HIVE-20590:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942403/HIVE-20590.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15026 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14243/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14243/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14243/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942403 - PreCommit-HIVE-Build
> Allow merge statement to have column schema
> ---
>
> Key: HIVE-20590
> URL: https://issues.apache.org/jira/browse/HIVE-20590
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Vineet Garg
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20590.01.patch, HIVE-20590.03.patch,
> HIVE-20590.04.patch
>
>
> Currently MERGE statement doesn't let user specify column schema with INSERT
> statements, therefore DEFAULT constraint are not applicable with it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20590) Allow merge statement to have column schema
[
https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639179#comment-16639179
]
Hive QA commented on HIVE-20590:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 617 unchanged - 13
fixed = 617 total (was 630) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} ql generated 0 new + 2319 unchanged - 1 fixed = 2319
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14243/dev-support/hive-personality.sh
|
| git revision | master / f0434c5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14243/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow merge statement to have column schema
> ---
>
> Key: HIVE-20590
> URL: https://issues.apache.org/jira/browse/HIVE-20590
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Vineet Garg
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-20590.01.patch, HIVE-20590.03.patch,
> HIVE-20590.04.patch
>
>
> Currently MERGE statement doesn't let user specify column schema with INSERT
> statements, therefore DEFAULT constraint are not applicable with it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639157#comment-16639157
]
Hive QA commented on HIVE-20661:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942377/HIVE-20661.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 15020 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=194)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPart (batchId=299)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial (batchId=299)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial2 (batchId=299)
org.apache.hadoop.hive.ql.TestTxnExIm.testExportPartPartial3 (batchId=299)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate (batchId=299)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportPartitionedCreate2 (batchId=299)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication
(batchId=244)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14242/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14242/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14242/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942377 - PreCommit-HIVE-Build
> Dynamic partitions loading calls add partition for every partition 1-by-1
> -
>
> Key: HIVE-20661
> URL: https://issues.apache.org/jira/browse/HIVE-20661
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20661.01.patch, HIVE-20661.02.patch,
> HIVE-20661.03.patch
>
>
> Hive.loadDynamicPartitions creates partitions using a threadpool, but the
> update of the Metastore via the MetastoreClient is done 1-by-1. This adds
> unnecessary extra calls. The partitions should be created in one batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Attachment: HIVE-20660.4.patch
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch, HIVE-20660.4.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639148#comment-16639148
]
Vineet Garg commented on HIVE-20660:
[~ashutoshc] Earlier patch had an issue, I have uploaded a new one, I'll create
RB as soon as I get a test run.
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch, HIVE-20660.4.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Status: Open (was: Patch Available)
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch, HIVE-20660.4.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20660:
---
Status: Patch Available (was: Open)
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch, HIVE-20660.4.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20687) Cancel Running Druid Query when a hive query is cancelled.
[
https://issues.apache.org/jira/browse/HIVE-20687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639083#comment-16639083
]
Ashutosh Chauhan commented on HIVE-20687:
-
Sounds like a good approach. One thing I am not sure is whether close() is
called on RecordReader when hive query is cancelled. I think we call close() on
tez job(dag) when query is canceled which internally might be calling close()
on RecordReader, but will be good to check.
> Cancel Running Druid Query when a hive query is cancelled.
> ---
>
> Key: HIVE-20687
> URL: https://issues.apache.org/jira/browse/HIVE-20687
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> https://issues.apache.org/jira/browse/HIVE-20686 ensures that hive query id
> is passed to druid.
> Druid also supports query cancellation by query id.
> Queries can be cancelled explicitly using their queryId by sending a DELETE
> request to following endpoint on the broker or router -
> {code}
> DELETE /druid/v2/{queryId}
> {code}
> Implementation wise we need to cancel a druid query when a RecordReader is
> closed.
> I believe Hive should already be calling close on open recordReaders when
> someone cancels a query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18785) Make JSON SerDe First-Class SerDe
[
https://issues.apache.org/jira/browse/HIVE-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639125#comment-16639125
]
Ashutosh Chauhan commented on HIVE-18785:
-
HIVE-20375 is a functional regression on master compared to branch-3. And then
there are many open items linked on this jira.
> Make JSON SerDe First-Class SerDe
> -
>
> Key: HIVE-18785
> URL: https://issues.apache.org/jira/browse/HIVE-18785
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Aihua Xu
>Priority: Major
>
> According to the [Hive SerDe
> Docs|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats],
> there are some extra steps involved in getting the JSON SerDe to work:
> {quote}
> ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> In some distributions, a reference to hive-hcatalog-core.jar is required.
> ADD JAR /usr/lib/hive-hcatalog/lib/hive-hcatalog-core.jar;
> {quote}
> I would like to propose that we move this SerDe into first-class status:
> {{STORED AS JSONFILE}}
> The user should have to perform no additional steps to use this SerDe.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20661) Dynamic partitions loading calls add partition for every partition 1-by-1
[
https://issues.apache.org/jira/browse/HIVE-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639121#comment-16639121
]
Hive QA commented on HIVE-20661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 6 new + 216 unchanged - 3
fixed = 222 total (was 219) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 1 new + 2320 unchanged - 0 fixed = 2321
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$2(Map,
Map) makes inefficient use of keySet iterator instead of entrySet iterator At
Hive.java:of keySet iterator instead of entrySet iterator At Hive.java:[line
2492] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14242/dev-support/hive-personality.sh
|
| git revision | master / f0434c5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14242/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14242/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14242/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamic partitions loading calls add partition for every partition 1-by-1
> -
>
> Key: HIVE-20661
> URL: https://issues.apache.org/jira/browse/HIVE-20661
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-20661.01.patch, HIVE-20661.02.patch,
> HIVE-20661.03.patch
>
>
> Hive.loadDynamicPartitions creates partitions using a threadpool, but the
> update of the Metastore via the MetastoreClient is done 1-by-1. This adds
> unnecessary extra calls. The partitions should be created in one batch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[
https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639096#comment-16639096
]
Ashutosh Chauhan commented on HIVE-20660:
-
[~vgarg] Can you update the golden file and create RB for this?
> Group by statistics estimation could be improved by bounding the total number
> of rows to source table
> -
>
> Key: HIVE-20660
> URL: https://issues.apache.org/jira/browse/HIVE-20660
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20660.1.patch, HIVE-20660.2.patch,
> HIVE-20660.3.patch
>
>
> Currently the stats for group by is estimated by taking product of NDVs of
> all the keys and bounding it by the number of rows of its input. This bound
> could be improved by using the source table instead of immediate input, the
> insight in this case is that cardinality/ndvs of a table can not go beyond
> the original (outer joins will only add NULLs thereby increasing the
> cardinality by 1).
> Note that the assumption here is that group by keys all belong to the same
> source table/input.
> This will improve the estimation in situations where group by is executed
> after joins wherein Hive could end up estimating the number of rows.
> *Reproducer*
> {code:sql}
> set hive.stats.fetch.column.stats=true;
> create table t1(i int, j int);
> alter table t1 update statistics set('numRows'='1',
> 'rawDataSize'='18000');
> alter table t1 update statistics for column i
> set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0');
> alter table t1 update statistics for column j
> set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50');
> create table t2(i2 int, j2 int);
> alter table t2 update statistics set('numRows'='1',
> 'rawDataSize'='1');
> alter table t2 update statistics for column i2
> set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0');
> alter table t2 update statistics for column j2
> set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1');
> explain select count (1) from t1,t2
> where t1.j=t2.i2
> group by t1.i, t1.j;
> {code}
> {code:sql}
> Reducer 2
> Reduce Operator Tree:
> Merge Join Operator
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col1 (type: int)
> 1 _col0 (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 99700 Data size: 797288 Basic stats:
> COMPLETE Column stats: COMPLETE
> Group By Operator
> aggregations: count()
> keys: _col0 (type: int), _col1 (type: int)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 49850 Data size: 797448 Basic stats:
> COMPLETE Column stats: COMPLETE <==
> Reduce Output Operator
> key expressions: _col0 (type: int), _col1 (type: int)
> sort order: ++
> Map-reduce partition columns: _col0 (type: int), _col1
> (type: int)
> Statistics: Num rows: 49850 Data size: 797448 Basic
> stats: COMPLETE Column stats: COMPLETE
> value expressions: _col2 (type: bigint)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639106#comment-16639106
]
Hive QA commented on HIVE-20647:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942360/HIVE-20647.9.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15025 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14241/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14241/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14241/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942360 - PreCommit-HIVE-Build
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch, HIVE-20647.8.patch, HIVE-20647.9.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
> (UR_UNINIT_READ_CALLED_FROM_SUPER_CONSTRUCTOR: Uninitialized read of field
> method called from constructor of superclass)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18785) Make JSON SerDe First-Class SerDe
[
https://issues.apache.org/jira/browse/HIVE-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639089#comment-16639089
]
Aihua Xu commented on HIVE-18785:
-
[~belugabehr] What kind of other tasks remaining for JSON work? Can we resolve
this now?
> Make JSON SerDe First-Class SerDe
> -
>
> Key: HIVE-18785
> URL: https://issues.apache.org/jira/browse/HIVE-18785
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Aihua Xu
>Priority: Major
>
> According to the [Hive SerDe
> Docs|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats],
> there are some extra steps involved in getting the JSON SerDe to work:
> {quote}
> ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> In some distributions, a reference to hive-hcatalog-core.jar is required.
> ADD JAR /usr/lib/hive-hcatalog/lib/hive-hcatalog-core.jar;
> {quote}
> I would like to propose that we move this SerDe into first-class status:
> {{STORED AS JSONFILE}}
> The user should have to perform no additional steps to use this SerDe.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20686) Sync QueryIDs across hive and druid
[ https://issues.apache.org/jira/browse/HIVE-20686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639085#comment-16639085 ] Ashutosh Chauhan commented on HIVE-20686: - Need to update Druid tests, others look like flaky test. > Sync QueryIDs across hive and druid > --- > > Key: HIVE-20686 > URL: https://issues.apache.org/jira/browse/HIVE-20686 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20686.patch > > > For the queries that hive passes to druid, pass on additional queryID as > query context. > It will be useful in tracing query level metrics across druid and hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20684) Analyze table compute stats fails for tables containing timestamp with local time zone column
[
https://issues.apache.org/jira/browse/HIVE-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639081#comment-16639081
]
Ashutosh Chauhan commented on HIVE-20684:
-
+1
If failures are not related, can you reattach patch to get a clean run.
> Analyze table compute stats fails for tables containing timestamp with local
> time zone column
> -
>
> Key: HIVE-20684
> URL: https://issues.apache.org/jira/browse/HIVE-20684
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20684.patch
>
>
> Analyze table druid_table compute statistics for columns;
> Reference Exception -
> {code}
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: Only
> integer/long/timestamp/date/float/double/string/binary/boolean/decimal
> type argument is accepted but timestamp with local time zone is passed.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats.getEvaluator(GenericUDAFComputeStats.java:105)
> at
> org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver.getEvaluator(AbstractGenericUDAFResolver.java:48)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1043)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:4817)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:5482)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:6496)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10617)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11557)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12229)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12319)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11802)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12254) Improve logging with yarn/hdfs
[ https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639082#comment-16639082 ] Aihua Xu commented on HIVE-12254: - [~pvary] Can you take a look at the latest patch? The test failure is not related. I ran twice and it failed with different ones. > Improve logging with yarn/hdfs > -- > > Key: HIVE-12254 > URL: https://issues.apache.org/jira/browse/HIVE-12254 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 1.2.1 >Reporter: Vikram Dixit K >Assignee: Aihua Xu >Priority: Major > Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch, > HIVE-12254.3.patch, HIVE-12254.4.patch, HIVE-12254.5.patch > > > In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both > HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS > issue) before this can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20590) Allow merge statement to have column schema
[ https://issues.apache.org/jira/browse/HIVE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639078#comment-16639078 ] Ashutosh Chauhan commented on HIVE-20590: - +1 pending tests. > Allow merge statement to have column schema > --- > > Key: HIVE-20590 > URL: https://issues.apache.org/jira/browse/HIVE-20590 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Vineet Garg >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-20590.01.patch, HIVE-20590.03.patch, > HIVE-20590.04.patch > > > Currently MERGE statement doesn't let user specify column schema with INSERT > statements, therefore DEFAULT constraint are not applicable with it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE
[
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639065#comment-16639065
]
Ashutosh Chauhan commented on HIVE-14431:
-
+1 pending tests
> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch,
> HIVE-14431.04.patch, HIVE-14431.05.patch, HIVE-14431.05.patch,
> HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '') = '') OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could
> be pushed to the TS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639063#comment-16639063
]
Hive QA commented on HIVE-20647:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} itests/util: The patch generated 0 new + 148
unchanged - 17 fixed = 148 total (was 165) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} itests/util generated 0 new + 48 unchanged - 4 fixed
= 48 total (was 52) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14241/dev-support/hive-personality.sh
|
| git revision | master / fe3a457 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util itests/hive-unit U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14241/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HadoopVer was ignored in QTestUtil
> --
>
> Key: HIVE-20647
> URL: https://issues.apache.org/jira/browse/HIVE-20647
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch,
> HIVE-20647.3.patch, HIVE-20647.4.patch, HIVE-20647.5.patch,
> HIVE-20647.6.patch, HIVE-20647.7.patch, HIVE-20647.8.patch, HIVE-20647.9.patch
>
>
> * Removed obsolete hadoopVer from QTestUtil
> * Cleaned up QTestUtil, QTestArgumentsBuilder
> * Refactored AccumuloTestSetup, AccumuloQTestUtil due to findbugs violation:
>
[jira] [Updated] (HIVE-20672) Logging thread in LlapTaskSchedulerService should report every fixed interval
[ https://issues.apache.org/jira/browse/HIVE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20672: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the patch! > Logging thread in LlapTaskSchedulerService should report every fixed interval > - > > Key: HIVE-20672 > URL: https://issues.apache.org/jira/browse/HIVE-20672 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20672.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[
https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639032#comment-16639032
]
Hive QA commented on HIVE-20646:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942231/HIVE-20646.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14238/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14238/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14238/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942231/HIVE-20646.02.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942231 - PreCommit-HIVE-Build
> Partition filter condition is not pushed down to metastore query if it has IS
> NOT NULL.
> ---
>
> Key: HIVE-20646
> URL: https://issues.apache.org/jira/browse/HIVE-20646
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: Metastore, Performance, pull-request-available
> Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch
>
>
> If the partition filter condition has "is not null" then the filter query
> isn't getting pushed to the SQL query in RDMBS.
> This slows down metastore api calls for getting list of partitions with
> filter condition.
> This condition gets added by optimizer in many cases so this is affecting
> many queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-20671:
--
Attachment: HIVE-20671.04.patch
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.04.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639035#comment-16639035
]
Hive QA commented on HIVE-20563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942358/HIVE-20563.09.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14240/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14240/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14240/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942358/HIVE-20563.09.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942358 - PreCommit-HIVE-Build
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch, HIVE-20563.08.patch,
> HIVE-20563.09.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
>
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[
https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639033#comment-16639033
]
Hive QA commented on HIVE-20531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942248/HIVE-20531.08.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14239/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14239/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14239/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942248/HIVE-20531.08.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942248 - PreCommit-HIVE-Build
> Repl load on cloud storage file system can skip redundant move or add
> partition tasks.
> --
>
> Key: HIVE-20531
> URL: https://issues.apache.org/jira/browse/HIVE-20531
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch,
> HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch,
> HIVE-20531.06.patch, HIVE-20531.07.patch, HIVE-20531.08.patch
>
>
> In replication load, both add partition and insert operations are handled
> through import. Import creates 3 major tasks. Copy, add partition and move.
> Copy does the copy of data from source location to staging directory. Then
> add partition (which runs in parallel to copy) creates the partition in meta
> store. Its a no op in case of insert and by the time this ddl task is
> executed for insert partition would be already present. The third operation
> is move. Which actually moves the file from staging directory to actual
> location. And then in case of insert it adds the insert event to notification
> table. It does this for add partition operation which is redundant as the
> event for add partition would have been written already by ddl task. With the
> optimization to copy directly to actual table location in S3, move task can
> be avoided for add partition operation replay and replay of insert need not
> create the add partition (ddl) task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20671) Hive Streaming has a broken dependency on metastore-server
[
https://issues.apache.org/jira/browse/HIVE-20671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639028#comment-16639028
]
Alexander Kolbasov commented on HIVE-20671:
---
Looks like it was caused by pom.xml changes - I am reverting them. Tests now
run fine for me locally.
> Hive Streaming has a broken dependency on metastore-server
> --
>
> Key: HIVE-20671
> URL: https://issues.apache.org/jira/browse/HIVE-20671
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-20671.02.patch, HIVE-20671.03.patch,
> HIVE-20671.04.patch, HIVE-20671.1.patch
>
>
> {{mvn -DskipTests -Pitests clean test-compile -DskipTests=true}} fails from
> the root directory of hive. This is sometimes run as part of the yetus tests
> depending on the modified files. The error is
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-streaming: Compilation failure: Compilation
> failure:
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[44,40]
> cannot find symbol
> [ERROR] symbol: class LockComponentBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR]
> /Users/jmarhuenda/workspace/hive/streaming/src/java/org/apache/hive/streaming/HiveStreamingConnection.java:[45,40]
> cannot find symbol
> [ERROR] symbol: class LockRequestBuilder
> [ERROR] location: package org.apache.hadoop.hive.metastore
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hive-streaming
> {code}
> I'm not sure why this would happen since {{test-compile}} works from
> {{hive/streaming}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[
https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639027#comment-16639027
]
Hive QA commented on HIVE-20678:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942357/HIVE-20678.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15026 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14237/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14237/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14237/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942357 - PreCommit-HIVE-Build
> HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure
> compatibility
>
>
> Key: HIVE-20678
> URL: https://issues.apache.org/jira/browse/HIVE-20678
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Hive, Spark
>Affects Versions: 4.0.0
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Major
> Attachments: HIVE-20678.1.patch
>
>
> This issue was caused by same reason described in
> https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems
> to be abandoned, so create this one to resolve the issue.
> HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will
> break the compatibility in spark's API when it is using hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20538) Allow to store a key value together with a transaction.
[ https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20538: --- Attachment: HIVE-20538.5.patch Status: Patch Available (was: Open) > Allow to store a key value together with a transaction. > --- > > Key: HIVE-20538 > URL: https://issues.apache.org/jira/browse/HIVE-20538 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore, Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, > HIVE-20538.2.patch, HIVE-20538.3.patch, HIVE-20538.4.patch, HIVE-20538.5.patch > > > This can be useful for example to know if a transaction has already happened. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639017#comment-16639017
]
Vihang Karajgaonkar commented on HIVE-20306:
v19 also renames {{GetPartitionsProjectSpec}} struct to
{{GetPartitionsProjectionSpec}} which I think is a better name. No other code
changes were involved.
> Implement projection spec for fetching only requested fields from partitions
>
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch,
> HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch,
> HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch,
> HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch,
> HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch,
> HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch,
> HIVE-20306.19.patch, HIVE-20306.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639013#comment-16639013
]
Eugene Koifman edited comment on HIVE-20635 at 10/4/18 11:11 PM:
-
A nit: {{LOG.info("findOriginalMinMaxKeys(): " + keyIntervalTmp);}} is only
logged {{if (minRowId >= maxRowId) {}}
would be better to log it always
otherwise LGTM
was (Author: ekoifman):
A nit: {{ LOG.info("findOriginalMinMaxKeys(): " + keyIntervalTmp);}} is
only logged {{if (minRowId >= maxRowId) {}}
would be better to log it always
otherwise LGTM
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.2.patch, HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: HIVE-20306.19.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.19.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20635) VectorizedOrcAcidRowBatchReader doesn't filter delete events for original files
[
https://issues.apache.org/jira/browse/HIVE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639013#comment-16639013
]
Eugene Koifman commented on HIVE-20635:
---
A nit: {{ LOG.info("findOriginalMinMaxKeys(): " + keyIntervalTmp);}} is
only logged {{if (minRowId >= maxRowId) {}}
would be better to log it always
otherwise LGTM
> VectorizedOrcAcidRowBatchReader doesn't filter delete events for original
> files
> ---
>
> Key: HIVE-20635
> URL: https://issues.apache.org/jira/browse/HIVE-20635
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Saurabh Seth
>Priority: Major
> Attachments: HIVE-20635.2.patch, HIVE-20635.patch
>
>
> this is a followup to HIVE-16812 which adds support for delete event
> filtering for splits from native acid files
> need to add the same for {{OrcSplit.isOriginal()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: (was: HIVE-20306.19.patch) > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.19.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20538) Allow to store a key value together with a transaction.
[ https://issues.apache.org/jira/browse/HIVE-20538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20538: --- Status: Open (was: Patch Available) > Allow to store a key value together with a transaction. > --- > > Key: HIVE-20538 > URL: https://issues.apache.org/jira/browse/HIVE-20538 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore, Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20538.1.patch, HIVE-20538.1.patch, > HIVE-20538.2.patch, HIVE-20538.3.patch, HIVE-20538.4.patch > > > This can be useful for example to know if a transaction has already happened. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17296) Acid tests with multiple splits
[ https://issues.apache.org/jira/browse/HIVE-17296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638973#comment-16638973 ] Eugene Koifman commented on HIVE-17296: --- Hive is now on ORC 1.5.3 > Acid tests with multiple splits > --- > > Key: HIVE-17296 > URL: https://issues.apache.org/jira/browse/HIVE-17296 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > > data files in an Acid table are ORC files which may have multiple stripes > for such files in base/ or delta/ (and original files with non acid to acid > conversion) are split by OrcInputFormat into multiple (stripe sized) chunks. > There is additional logic in in OrcRawRecordMerger > (discoverKeyBounds/discoverOriginalKeyBounds) that is not tested by any E2E > tests since none of the have enough data to generate multiple stripes in a > single file. > testRecordReaderOldBaseAndDelta/testRecordReaderNewBaseAndDelta/testOriginalReaderPair > in TestOrcRawRecordMerger has some logic to test this but it really needs e2e > tests. > With ORC-228 it will be possible to write such tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20678) HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure compatibility
[
https://issues.apache.org/jira/browse/HIVE-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638976#comment-16638976
]
Hive QA commented on HIVE-20678:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-handler: The patch generated 8 new + 6 unchanged
- 1 fixed = 14 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} hbase-handler generated 0 new + 7 unchanged - 6
fixed = 7 total (was 13) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14237/dev-support/hive-personality.sh
|
| git revision | master / fe3a457 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14237/yetus/diff-checkstyle-hbase-handler.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14237/yetus/patch-asflicense-problems.txt
|
| modules | C: hbase-handler U: hbase-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14237/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveHBaseTableOutputFormat should implement HiveOutputFormat to ensure
> compatibility
>
>
> Key: HIVE-20678
> URL: https://issues.apache.org/jira/browse/HIVE-20678
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler, Hive, Spark
>Affects Versions: 4.0.0
>Reporter: Alice Fan
>Assignee: Alice Fan
>Priority: Major
> Attachments: HIVE-20678.1.patch
>
>
> This issue was caused by same reason described in
> https://issues.apache.org/jira/browse/HIVE-11166. As the original Jira seems
> to be abandoned, so create this one to resolve the issue.
> HiveHBaseTableOutputFormat does not implemented HiveOutputFormat, it will
> break the compatibility in spark's API when it is using hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20664) Potential ArrayIndexOutOfBoundsException in VectorizedOrcAcidRowBatchReader.findMinMaxKeys
[
https://issues.apache.org/jira/browse/HIVE-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638970#comment-16638970
]
Eugene Koifman commented on HIVE-20664:
---
+1 on patch 2
regarding tests, take a look at ORC-228. The idea was precisely to enable
ability to create small stripes for tests.
Hive is now on Orc 1.5.3.
Let me know if you want to do that as part of this ticket or I can commit this
and you can do that in a follow up jira
> Potential ArrayIndexOutOfBoundsException in
> VectorizedOrcAcidRowBatchReader.findMinMaxKeys
> --
>
> Key: HIVE-20664
> URL: https://issues.apache.org/jira/browse/HIVE-20664
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Saurabh Seth
>Assignee: Saurabh Seth
>Priority: Minor
> Attachments: HIVE-20664.2.patch, HIVE-20664.patch
>
>
> [~ekoifman], could you please confirm if my understanding is correct and if
> so, review the fix?
> In the method {{VectorizedOrcAcidRowBatchReader.findMinMaxKeys}}, the code
> snippet that identifies the first and last stripe indices in the current
> split could result in an ArrayIndexOutOfBoundsException if a complete split
> is within the same stripe:
> {noformat}
> for(int i = 0; i < stripes.size(); i++) {
> StripeInformation stripe = stripes.get(i);
> long stripeEnd = stripe.getOffset() + stripe.getLength();
> if(firstStripeIndex == -1 && stripe.getOffset() >= splitStart) {
> firstStripeIndex = i;
> }
> if(lastStripeIndex == -1 && splitEnd <= stripeEnd &&
> stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset() ) {
> //the last condition is for when both splitStart and splitEnd are in
> // the same stripe
> lastStripeIndex = i;
> }
> }
> {noformat}
> Consider the example where there are 2 stripes - 0-500 and 500-1000 and
> splitStart is 600 and splitEnd is 800.
> In the first iteration of the loop, stripe.getOffset() is 0 and stripeEnd is
> 500. In this iteration, neither of the if statement conditions will be met
> and firstSripeIndex as well as lastStripeIndex remain -1.
> In the second iteration of the loop stripe.getOffset() is 500, stripeEnd is
> 1000, The first if statement condition will not be met in this case because
> stripe's offset (500) is not greater than or equal to the splitStart (600).
> However, in the second if statement, splitEnd (800) is <= stripeEnd(1000) and
> it will try to compute the last condition
> {{stripes.get(firstStripeIndex).getOffset() <= stripe.getOffset()}}. This
> will throw an ArrayIndexOutOfBoundsException because firstStripeIndex is
> still -1.
> I'm not sure if this scenario is possible at all, hence logging this as a low
> priority issue. Perhaps block based split generation using BISplitStrategy
> could trigger this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17296) Acid tests with multiple splits
[ https://issues.apache.org/jira/browse/HIVE-17296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17296: -- Priority: Blocker (was: Critical) > Acid tests with multiple splits > --- > > Key: HIVE-17296 > URL: https://issues.apache.org/jira/browse/HIVE-17296 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > > data files in an Acid table are ORC files which may have multiple stripes > for such files in base/ or delta/ (and original files with non acid to acid > conversion) are split by OrcInputFormat into multiple (stripe sized) chunks. > There is additional logic in in OrcRawRecordMerger > (discoverKeyBounds/discoverOriginalKeyBounds) that is not tested by any E2E > tests since none of the have enough data to generate multiple stripes in a > single file. > testRecordReaderOldBaseAndDelta/testRecordReaderNewBaseAndDelta/testOriginalReaderPair > in TestOrcRawRecordMerger has some logic to test this but it really needs e2e > tests. > With ORC-228 it will be possible to write such tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14431) Recognize COALESCE as CASE
[
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14431:
---
Attachment: HIVE-14431.05.patch
> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch,
> HIVE-14431.04.patch, HIVE-14431.05.patch, HIVE-14431.05.patch,
> HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '') = '') OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could
> be pushed to the TS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638947#comment-16638947
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942355/HIVE-20679.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14236/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-10-04 22:06:43.647
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14236/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-10-04 22:06:43.650
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
97f0513..fe3a457 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 97f0513 HIVE-20563: Vectorization: CASE WHEN expression fails
when THEN/ELSE type and result type are different (Matt McCline, reviewed by
Teddy Choi)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at fe3a457 HIVE-20556: Expose an API to retrieve the TBL_ID from
TBLS in the metastore tables (Jaume Marhuenda via Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-10-04 22:06:45.160
+ rm -rf ../yetus_PreCommit-HIVE-Build-14236
+ mkdir ../yetus_PreCommit-HIVE-Build-14236
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14236
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14236/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 2724
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:514
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java'
cleanly.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java:1
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageBuilder.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java:20
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:514
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java'
cleanly.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java:1
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageBuilder.java'
with
[jira] [Commented] (HIVE-20586) Beeline is asking for user/pass when invoked without -u
[
https://issues.apache.org/jira/browse/HIVE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638944#comment-16638944
]
Hive QA commented on HIVE-20586:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942356/HIVE-20586.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15016 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=45)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14235/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14235/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14235/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942356 - PreCommit-HIVE-Build
> Beeline is asking for user/pass when invoked without -u
> ---
>
> Key: HIVE-20586
> URL: https://issues.apache.org/jira/browse/HIVE-20586
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Daniel Voros
>Assignee: Janos Gub
>Priority: Major
> Attachments: HIVE-20586.1.patch, HIVE-20586.1.patch, HIVE-20586.patch
>
>
> Since HIVE-18963 it's possible to define a default connection URL in
> beeline-site.xml to be able to use beeline without specifying the HS2 JDBC
> URL.
> When invoked with no arguments, beeline is asking for username/password on
> the command line. When running with {{-u}} and the exact same URL as in
> beeline-site.xml, it does not ask for username/password.
> I think these two should do exactly the same, given that the URL after {{-u}}
> is the same as in beeline-site.xml:
> {code:java}
> beeline -u URL
> beeline
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20556: -- Resolution: Fixed Fix Version/s: 4.0.0 Release Note: n/a Status: Resolved (was: Patch Available) committed to master thanks Jaume for the contribution > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20556.1.patch, HIVE-20556.10.patch, > HIVE-20556.11.patch, HIVE-20556.12.patch, HIVE-20556.13.patch, > HIVE-20556.14.patch, HIVE-20556.15.patch, HIVE-20556.16.patch, > HIVE-20556.17.patch, HIVE-20556.18.patch, HIVE-20556.2.patch, > HIVE-20556.3.patch, HIVE-20556.4.patch, HIVE-20556.5.patch, > HIVE-20556.6.patch, HIVE-20556.7.patch, HIVE-20556.8.patch, HIVE-20556.9.patch > > > We have two options to do this > 1) Use the current MTable and add a field for this value > 2) Add an independent API call to the metastore that would return the TBL_ID. > Option 1 is preferable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638900#comment-16638900 ] Vihang Karajgaonkar commented on HIVE-20306: The test failures are unrelated. They are working for me when I run with my patch. > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.19.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[ https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20306: --- Attachment: HIVE-20306.19.patch > Implement projection spec for fetching only requested fields from partitions > > > Key: HIVE-20306 > URL: https://issues.apache.org/jira/browse/HIVE-20306 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch, > HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch, > HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch, > HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch, > HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch, > HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch, > HIVE-20306.19.patch, HIVE-20306.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[ https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20291: --- Attachment: HIVE-20291.9.patch Status: Patch Available (was: Open) > Allow HiveStreamingConnection to receive a WriteId > -- > > Key: HIVE-20291 > URL: https://issues.apache.org/jira/browse/HIVE-20291 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch, > HIVE-20291.3.patch, HIVE-20291.4.patch, HIVE-20291.5.patch, > HIVE-20291.6.patch, HIVE-20291.7.patch, HIVE-20291.8.patch, HIVE-20291.9.patch > > > If the writeId is received externally it won't need to open connections to > the metastore. It won't be able to the commit in this case as well so it must > be done by the entity passing the writeId. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20291) Allow HiveStreamingConnection to receive a WriteId
[ https://issues.apache.org/jira/browse/HIVE-20291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20291: --- Status: Open (was: Patch Available) > Allow HiveStreamingConnection to receive a WriteId > -- > > Key: HIVE-20291 > URL: https://issues.apache.org/jira/browse/HIVE-20291 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20291.1.patch, HIVE-20291.2.patch, > HIVE-20291.3.patch, HIVE-20291.4.patch, HIVE-20291.5.patch, > HIVE-20291.6.patch, HIVE-20291.7.patch, HIVE-20291.8.patch > > > If the writeId is received externally it won't need to open connections to > the metastore. It won't be able to the commit in this case as well so it must > be done by the entity passing the writeId. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20586) Beeline is asking for user/pass when invoked without -u
[
https://issues.apache.org/jira/browse/HIVE-20586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638885#comment-16638885
]
Hive QA commented on HIVE-20586:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} beeline in master has 53 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 4
unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14235/dev-support/hive-personality.sh
|
| git revision | master / 97f0513 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14235/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: beeline itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14235/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Beeline is asking for user/pass when invoked without -u
> ---
>
> Key: HIVE-20586
> URL: https://issues.apache.org/jira/browse/HIVE-20586
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Daniel Voros
>Assignee: Janos Gub
>Priority: Major
> Attachments: HIVE-20586.1.patch, HIVE-20586.1.patch, HIVE-20586.patch
>
>
> Since HIVE-18963 it's possible to define a default connection URL in
> beeline-site.xml to be able to use beeline without specifying the HS2 JDBC
> URL.
> When invoked with no arguments, beeline is asking for username/password on
> the command line. When running with {{-u}} and the exact same URL as in
> beeline-site.xml, it does not ask for username/password.
> I think these two should do exactly the same, given that the URL after {{-u}}
> is the same as in beeline-site.xml:
> {code:java}
> beeline -u URL
> beeline
>
[jira] [Commented] (HIVE-20680) hive bootstrap missing partitions in replicated db
[
https://issues.apache.org/jira/browse/HIVE-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638848#comment-16638848
]
Hive QA commented on HIVE-20680:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942352/HIVE-20680.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15015 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSparkSessionTimeout
(batchId=246)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942352 - PreCommit-HIVE-Build
> hive bootstrap missing partitions in replicated db
> --
>
> Key: HIVE-20680
> URL: https://issues.apache.org/jira/browse/HIVE-20680
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20680.01.patch
>
>
> The issue is
> 1. When bootstrap was going on, kill query was called to kill the repl load
> command.
> 2. During restart, one table with no partition set the scope to table as the
> ckpt property was not yet set for that table.
> 3. Due to this, all partitioned table after this didn't get their tasks
> related to partition appended in the root task if the ckpt property is
> already updated for those tables.
> The fix is to reset the table scope to false if for that table there are no
> tasks added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20403) Add blobstore tests for ALTER TABLE CONCATENATE and MSCK REPAIR TABLE
[ https://issues.apache.org/jira/browse/HIVE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyang Li updated HIVE-20403: -- Attachment: HIVE-20403(1).patch HIVE-20403(2).patch Status: Patch Available (was: Open) > Add blobstore tests for ALTER TABLE CONCATENATE and MSCK REPAIR TABLE > - > > Key: HIVE-20403 > URL: https://issues.apache.org/jira/browse/HIVE-20403 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Junyang Li >Assignee: Junyang Li >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-20403(1).patch, HIVE-20403(1).patch, > HIVE-20403(2).patch, HIVE-20403(2).patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: alter_table_concatenate.q and recover_partitions.q. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19408) Improve show materialized views statement to show more information about invalidation
[ https://issues.apache.org/jira/browse/HIVE-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19408: -- Hadoop Flags: Incompatible change > Improve show materialized views statement to show more information about > invalidation > - > > Key: HIVE-19408 > URL: https://issues.apache.org/jira/browse/HIVE-19408 > Project: Hive > Issue Type: Sub-task > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19408.01.patch, HIVE-19408.02.patch, > HIVE-19408.03.patch, HIVE-19408.patch > > > We should show more useful information in addition to materialized view name. > For instance, information about whether the materialized view contents are > up-to-date or not, and which table(s) have changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14431) Recognize COALESCE as CASE
[
https://issues.apache.org/jira/browse/HIVE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638810#comment-16638810
]
Ashutosh Chauhan commented on HIVE-14431:
-
HIVE-20563 is in. [~jcamachorodriguez] Can you reattach this patch?
> Recognize COALESCE as CASE
> --
>
> Key: HIVE-14431
> URL: https://issues.apache.org/jira/browse/HIVE-14431
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-14431.01.patch, HIVE-14431.03.patch,
> HIVE-14431.04.patch, HIVE-14431.05.patch, HIVE-14431.2.patch, HIVE-14431.patch
>
>
> Transform:
> {code}
> (COALESCE(a, '') = '') OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> into:
> {code}
> (a='') OR
>(a is null) OR
>(a = 'A' AND b = c) OR
>(a = 'B' AND b = d) OR
>(a = 'C' AND b = e) OR
>(a = 'D' AND b = f) OR
>(a = 'E' AND b = g) OR
>(a = 'F' AND b = h)
> {code}
> With complex queries, this will lead us to factor more predicates that could
> be pushed to the TS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20658) "show tables" should show view as well
[ https://issues.apache.org/jira/browse/HIVE-20658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20658: -- Resolution: Invalid Status: Resolved (was: Patch Available) Close the issue as this is by design. But we shall highlight this as a backward incompatibility change. > "show tables" should show view as well > -- > > Key: HIVE-20658 > URL: https://issues.apache.org/jira/browse/HIVE-20658 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20658.1.patch > > > "show tables" changed behavior to show real table (no view) in HIVE-19408. > This breaks backward compatibility and we shall restore default behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20680) hive bootstrap missing partitions in replicated db
[
https://issues.apache.org/jira/browse/HIVE-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638807#comment-16638807
]
Hive QA commented on HIVE-20680:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
41s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 119
unchanged - 0 fixed = 129 total (was 119) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14234/dev-support/hive-personality.sh
|
| git revision | master / 97f0513 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14234/yetus/patch-mvninstall-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14234/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: common itests/hive-unit ql standalone-metastore/metastore-server
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14234/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> hive bootstrap missing partitions in replicated db
> --
>
> Key: HIVE-20680
> URL: https://issues.apache.org/jira/browse/HIVE-20680
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
[jira] [Updated] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-20563:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch, HIVE-20563.08.patch,
> HIVE-20563.09.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
>
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638751#comment-16638751
]
Matt McCline commented on HIVE-20563:
-
Committed to master. [~teddy.choi] thank you for your review!
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch, HIVE-20563.08.patch,
> HIVE-20563.09.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> cstring1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:965)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:938)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638740#comment-16638740
]
Hive QA commented on HIVE-20563:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942358/HIVE-20563.09.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15015 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14233/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14233/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14233/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942358 - PreCommit-HIVE-Build
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch, HIVE-20563.08.patch,
> HIVE-20563.09.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:973)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
>
[jira] [Commented] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638726#comment-16638726 ] Eugene Koifman commented on HIVE-20556: --- +1 patch 18 > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20556.1.patch, HIVE-20556.10.patch, > HIVE-20556.11.patch, HIVE-20556.12.patch, HIVE-20556.13.patch, > HIVE-20556.14.patch, HIVE-20556.15.patch, HIVE-20556.16.patch, > HIVE-20556.17.patch, HIVE-20556.18.patch, HIVE-20556.2.patch, > HIVE-20556.3.patch, HIVE-20556.4.patch, HIVE-20556.5.patch, > HIVE-20556.6.patch, HIVE-20556.7.patch, HIVE-20556.8.patch, HIVE-20556.9.patch > > > We have two options to do this > 1) Use the current MTable and add a field for this value > 2) Add an independent API call to the metastore that would return the TBL_ID. > Option 1 is preferable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20563) Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type are different
[
https://issues.apache.org/jira/browse/HIVE-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638688#comment-16638688
]
Hive QA commented on HIVE-20563:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 11 new + 778 unchanged - 5
fixed = 789 total (was 783) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} ql generated 0 new + 2320 unchanged - 1 fixed = 2320
total (was 2321) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14233/dev-support/hive-personality.sh
|
| git revision | master / 857259e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14233/yetus/diff-checkstyle-ql.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14233/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: CASE WHEN expression fails when THEN/ELSE type and result type
> are different
> ---
>
> Key: HIVE-20563
> URL: https://issues.apache.org/jira/browse/HIVE-20563
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Matt McCline
>Priority: Major
> Attachments: HIVE-20563.01.patch, HIVE-20563.02.patch,
> HIVE-20563.03.patch, HIVE-20563.04.patch, HIVE-20563.05.patch,
> HIVE-20563.06.patch, HIVE-20563.07.patch, HIVE-20563.08.patch,
> HIVE-20563.09.patch
>
>
> With the following stacktrace:
> {code}
> java.lang.Exception: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
> ~[hadoop-mapreduce-client-common-3.1.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
> [hadoop-mapreduce-client-common-3.1.0.jar:?]
> Caused by:
[jira] [Updated] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20652:
---
Attachment: HIVE-20652.02.patch
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch, HIVE-20652.02.patch,
> HIVE-20652.patch, external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:295)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:277)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genTablePlan(SemanticAnalyzer.java:11100)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11468)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11427)
> [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:525)
>
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638636#comment-16638636
]
Hive QA commented on HIVE-20306:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942340/HIVE-20306.18.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15046 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_context_ngrams]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_histogram_numeric]
(batchId=14)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14232/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14232/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14232/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942340 - PreCommit-HIVE-Build
> Implement projection spec for fetching only requested fields from partitions
>
>
> Key: HIVE-20306
> URL: https://issues.apache.org/jira/browse/HIVE-20306
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20306.02.patch, HIVE-20306.03.patch,
> HIVE-20306.04.patch, HIVE-20306.05.patch, HIVE-20306.06.patch,
> HIVE-20306.07.patch, HIVE-20306.08.patch, HIVE-20306.09.patch,
> HIVE-20306.10.patch, HIVE-20306.11.patch, HIVE-20306.12.patch,
> HIVE-20306.13.patch, HIVE-20306.14.patch, HIVE-20306.15.patch,
> HIVE-20306.16.patch, HIVE-20306.17.patch, HIVE-20306.18.patch,
> HIVE-20306.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20545) Ability to exclude potentially large parameters in HMS Notifications
[ https://issues.apache.org/jira/browse/HIVE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638616#comment-16638616 ] Andrew Sherman commented on HIVE-20545: --- Pushed to master, thanks [~bharos92] > Ability to exclude potentially large parameters in HMS Notifications > > > Key: HIVE-20545 > URL: https://issues.apache.org/jira/browse/HIVE-20545 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20545.1.patch, HIVE-20545.2.patch, > HIVE-20545.3.branch-3.patch, HIVE-20545.3.patch, HIVE-20545.4.patch, > HIVE-20545.6.patch, HIVE-20545.7.patch > > > Clients can add large-sized parameters in Table/Partition objects. So we need > to enable adding regex patterns through HiveConf to match parameters to be > filtered from table and partition objects before serialization in HMS > notifications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20306) Implement projection spec for fetching only requested fields from partitions
[
https://issues.apache.org/jira/browse/HIVE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638594#comment-16638594
]
Hive QA commented on HIVE-20306:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 53 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 179 unchanged - 3 fixed = 181 total (was 182) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlBoolean(Object)
has Boolean return type and returns explicit null At
MetastoreDirectSqlUtils.java:and returns explicit null At
MetastoreDirectSqlUtils.java:[line 514] |
| | org.apache.hadoop.hive.metastore.ObjectStore$16.run(List) concatenates
strings using + in a loop At ObjectStore.java:in a loop At
ObjectStore.java:[line 8727] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14232/dev-support/hive-personality.sh
|
| git revision | master / d0ed25e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14232/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14232/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14232/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14232/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-common itests/hcatalog-unit
standalone-metastore/metastore-server U: . |
| Console output |
[jira] [Commented] (HIVE-20652) JdbcStorageHandler push join of two different datasource to jdbc driver
[
https://issues.apache.org/jira/browse/HIVE-20652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638540#comment-16638540
]
Hive QA commented on HIVE-20652:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942336/HIVE-20652.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14231/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14231/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14231/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942336/HIVE-20652.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942336 - PreCommit-HIVE-Build
> JdbcStorageHandler push join of two different datasource to jdbc driver
> ---
>
> Key: HIVE-20652
> URL: https://issues.apache.org/jira/browse/HIVE-20652
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Daniel Dai
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20652.01.patch, HIVE-20652.patch,
> external_jdbc_table2.q
>
>
> Test case attached. The following query fail:
> {code}
> SELECT * FROM ext_auth1 JOIN ext_auth2 ON ext_auth1.ikey = ext_auth2.ikey
> {code}
> Error message:
> {code}
> 2018-09-28T00:36:23,860 DEBUG [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Query to execute is [SELECT *
> FROM (SELECT *
> FROM "SIMPLE_DERBY_TABLE1"
> WHERE "ikey" IS NOT NULL) AS "t"
> INNER JOIN (SELECT *
> FROM "SIMPLE_DERBY_TABLE2"
> WHERE "ikey" IS NOT NULL) AS "t0" ON "t"."ikey" = "t0"."ikey" {LIMIT 1}]
> 2018-09-28T00:36:23,864 ERROR [17b954d9-3250-45a9-995e-1b3f8277a681 main]
> dao.GenericJdbcDatabaseAccessor: Error while trying to get column names.
> java.sql.SQLSyntaxErrorException: Table/View 'SIMPLE_DERBY_TABLE2' does not
> exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown
> Source) ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
> ~[derby-10.14.1.0.jar:?]
> at
> org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:281)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:313)
> ~[commons-dbcp-1.4.jar:1.4]
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getColumnNames(GenericJdbcDatabaseAccessor.java:74)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hive.storage.jdbc.JdbcSerDe.initialize(JdbcSerDe.java:78)
> [hive-jdbc-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540)
> [hive-serde-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
> [hive-metastore-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-12254) Improve logging with yarn/hdfs
[
https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638536#comment-16638536
]
Hive QA commented on HIVE-12254:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942326/HIVE-12254.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15010 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
(batchId=154)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14229/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14229/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14229/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942326 - PreCommit-HIVE-Build
> Improve logging with yarn/hdfs
> --
>
> Key: HIVE-12254
> URL: https://issues.apache.org/jira/browse/HIVE-12254
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 1.2.1
>Reporter: Vikram Dixit K
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch,
> HIVE-12254.3.patch, HIVE-12254.4.patch, HIVE-12254.5.patch
>
>
> In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both
> HIVE-12249 and HDFS-9184 are required (and upgraded in hive for the HDFS
> issue) before this can be resolved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17043) Remove non unique columns from group by keys if not referenced later
[
https://issues.apache.org/jira/browse/HIVE-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638538#comment-16638538
]
Hive QA commented on HIVE-17043:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942306/HIVE-17043.9.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14230/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14230/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14230/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12942306/HIVE-17043.9.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942306 - PreCommit-HIVE-Build
> Remove non unique columns from group by keys if not referenced later
>
>
> Key: HIVE-17043
> URL: https://issues.apache.org/jira/browse/HIVE-17043
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-17043.1.patch, HIVE-17043.2.patch,
> HIVE-17043.3.patch, HIVE-17043.4.patch, HIVE-17043.5.patch,
> HIVE-17043.6.patch, HIVE-17043.7.patch, HIVE-17043.8.patch, HIVE-17043.9.patch
>
>
> Group by keys may be a mix of unique (or primary) keys and regular columns.
> In such cases presence of regular column won't alter cardinality of groups.
> So, if regular columns are not referenced later, they can be dropped from
> group by keys. Depending on operator tree may result in those columns not
> being read at all from disk in best case. In worst case, we will avoid
> shuffling and sorting regular columns from mapper to reducer, which still
> could be substantial CPU and network savings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20544) TOpenSessionReq logs password and username
[
https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-20544:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~klcopp] and [~asherman] for the review!
> TOpenSessionReq logs password and username
> --
>
> Key: HIVE-20544
> URL: https://issues.apache.org/jira/browse/HIVE-20544
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: beginner, patch, security
> Fix For: 4.0.0
>
> Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch,
> HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch,
> HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch,
> HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.patch, non-solution.patch,
> working-solution.patch
>
>
> In
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq,
> if client protocol is unset, validate() and toString() prints both username
> and password to logs.
> Logging a password is a security risk. We should hide the ***.
> =Edit= (no longer relevant, see comments)
> This issue is tricky since it is caused in a fully generated class. I've been
> playing around and have found one working solution, butI'd truly appreciate
> ideas for a more elegant solution or input.
> The problem:
> TCLIService.thrift is the template for generating all classes in
> service-rpc. Struct TOpenSessionReq is OpenSession()'s one parameter and is
> defined thus:
> {noformat}
> struct TOpenSessionReq {
> 1: required TProtocolVersion client_protocol =
> TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10
> 2: optional string username
> 3: optional string password
> 4: optional map configuration
> }
> {noformat}
> In the generated class TOpenSessionReq.java, client_protocol is checked by a
> validate() method, which is called quite a few times; if client_protocol is
> not set, it throws a TProtocolException, passing along a toString(). This
> toString() gets the names and values of all fields, including username and
> password.
> Working solution:
> * Create a separate struct containing only the username and password, and
> pass it to OpenSession() as a second parameter. Since all fields in the new
> struct are "optional", the generated validate() is empty – toString() is
> never used. This involves changing core classes and breaks the "Each function
> should take exactly one parameter" coding convention (detailed at
> service-rpc/if/TCLIService.thrift:27).
> See working-solution.patch.
> What doesn't work:
> * Making client_protocol optional instead of required. Apparently this will
> break everything.
> * Overwriting toString() – TOpenSessionReq is a struct.
> * Creating two Thrift structs, one struct for required (TRequiredReq) and
> one for optional (TOptionalReq) fields, and nesting them in struct
> TOpenSessionReq. This doesn't work because validate() in TOpenSessionReq can
> call TOptionalReq.toString(), which prints the password to logs. This will
> happen if TRequiredReq.client_protocol isn't set.
> See non-solution.patch
> * Asking Thrift devs to change their code. I wrote them an email but have no
> expectations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20670) ShuffleHandler should use HiveConf.LLAP_DAEMON_YARN_SHUFFLE_PORT instead of local variable
[ https://issues.apache.org/jira/browse/HIVE-20670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20670: --- Attachment: HIVE-20670.2.patch Status: Patch Available (was: Open) > ShuffleHandler should use HiveConf.LLAP_DAEMON_YARN_SHUFFLE_PORT instead of > local variable > -- > > Key: HIVE-20670 > URL: https://issues.apache.org/jira/browse/HIVE-20670 > Project: Hive > Issue Type: Bug > Components: HiveServer2, llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20670.1.patch, HIVE-20670.2.patch, > HIVE-20670.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20670) ShuffleHandler should use HiveConf.LLAP_DAEMON_YARN_SHUFFLE_PORT instead of local variable
[ https://issues.apache.org/jira/browse/HIVE-20670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20670: --- Status: Open (was: Patch Available) > ShuffleHandler should use HiveConf.LLAP_DAEMON_YARN_SHUFFLE_PORT instead of > local variable > -- > > Key: HIVE-20670 > URL: https://issues.apache.org/jira/browse/HIVE-20670 > Project: Hive > Issue Type: Bug > Components: HiveServer2, llap >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20670.1.patch, HIVE-20670.2.patch, > HIVE-20670.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-17300: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Finally pushed to master. Thanks for all your work [~klcopp] and for the review [~szehon]! > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, features, patch > Fix For: 4.0.0 > > Attachments: HIVE-17300.10.patch, HIVE-17300.10.patch, > HIVE-17300.10.patch, HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.6.patch, HIVE-17300.7.patch, > HIVE-17300.7.patch, HIVE-17300.8.patch, HIVE-17300.8.patch, > HIVE-17300.8.patch, HIVE-17300.8.patch, HIVE-17300.9.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12254) Improve logging with yarn/hdfs
[
https://issues.apache.org/jira/browse/HIVE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638470#comment-16638470
]
Hive QA commented on HIVE-12254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
19s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2321 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} cli in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14229/dev-support/hive-personality.sh
|
| git revision | master / e96ea66 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14229/yetus/patch-mvninstall-cli.txt
|
| modules | C: shims/common shims/0.23 cli ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14229/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve logging with yarn/hdfs
> --
>
> Key: HIVE-12254
> URL: https://issues.apache.org/jira/browse/HIVE-12254
> Project: Hive
> Issue Type: Bug
> Components: Shims
>Affects Versions: 1.2.1
>Reporter: Vikram Dixit K
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-12254.1.patch, HIVE-12254.2.patch,
> HIVE-12254.3.patch, HIVE-12254.4.patch, HIVE-12254.5.patch
>
>
> In extension to HIVE-12249, adding info for Yarn/HDFS as well. Both
> HIVE-12249 and HDFS-9184 are required (and upgraded
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20535: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks for all your work [~dkuzmenko] and [~kgyrtkirk] for the review! > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch, > HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch, > HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch, > HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch, > HIVE-20535.2.patch, HIVE-20535.20.patch, HIVE-20535.21.patch, > HIVE-20535.22.patch, HIVE-20535.23.patch, HIVE-20535.3.patch, > HIVE-20535.4.patch, HIVE-20535.5.patch, HIVE-20535.6.patch, > HIVE-20535.8.patch, HIVE-20535.9.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20691: --- Issue Type: Test (was: Bug) > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Test >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20691.patch > > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-20691. Resolution: Fixed Fix Version/s: 4.0.0 Thanks [~kgyrtkirk] > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20691.patch > > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638465#comment-16638465 ] Zoltan Haindrich commented on HIVE-20691: - +1 > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20691.patch > > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20691: --- Attachment: HIVE-20691.patch > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20691.patch > > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638457#comment-16638457 ] Jesus Camacho Rodriguez commented on HIVE-20691: [~kgyrtkirk], [~ashutoshc], [~vgarg], can you review? Thanks > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20691 started by Jesus Camacho Rodriguez. -- > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20691) Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl]
[ https://issues.apache.org/jira/browse/HIVE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20691: -- > Fix org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cttl] > > > Key: HIVE-20691 > URL: https://issues.apache.org/jira/browse/HIVE-20691 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Blocking all ptest runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Status: Open (was: Patch Available) Cancelling patch to address Thejas's comments. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Attachment: HIVE-20644.04 Status: Patch Available (was: Open) Attaching patch with Thejas's comments addressed. I ran the three testcases which failed with the previous patch separately. All of those passed on my machine. Also I observe that the diffs are unrelated to the patch. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02, HIVE-20644.03, > HIVE-20644.04 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
