[jira] [Assigned] (HIVE-21260) Hive 3 (onprem) -> 4(onprem): Hive replication failed due to postgres sql execution issue
[
https://issues.apache.org/jira/browse/HIVE-21260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera reassigned HIVE-21260:
--
> Hive 3 (onprem) -> 4(onprem): Hive replication failed due to postgres sql
> execution issue
> -
>
> Key: HIVE-21260
> URL: https://issues.apache.org/jira/browse/HIVE-21260
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
>
> Missing quotes in sql string is causing sql execution error for postgres.
>
> {code:java}
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) -
> MetaException(message:Unable to update transaction database
> org.postgresql.util.PSQLException: ERROR: relat
> ion "database_params" does not exist
> Position: 25
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2284)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2003)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:321)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:284)
> at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.updateReplId(TxnHandler.java:907)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(TxnHandler.java:1023)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.commit_txn(HiveMetaStore.java:7703)
> at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy39.commit_txn(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:18730)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$commit_txn.getResult(ThriftHiveMetastore.java:18714)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> ){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766886#comment-16766886
]
Hive QA commented on HIVE-21240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} serde: The patch generated 5 new + 20 unchanged - 9
fixed = 25 total (was 29) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 2 new + 6 unchanged - 5 fixed
= 8 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} serde generated 1 new + 193 unchanged - 4 fixed = 194
total (was 197) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.serde2.json.HiveJsonReader.getColIndex(String) At
HiveJsonReader.java:org.apache.hadoop.hive.serde2.json.HiveJsonReader.getColIndex(String)
At HiveJsonReader.java:[line 484] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16047/dev-support/hive-personality.sh
|
| git revision | master / 5db4c77 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16047/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16047/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16047/yetus/new-findbugs-serde.html
|
| modules | C: serde ql hcatalog/core U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16047/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
>
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766867#comment-16766867 ] Prasanth Jayachandran commented on HIVE-21254: -- Doesn't look like related test failure. Retrying. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.4.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch, HIVE-21254.4.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766854#comment-16766854
]
Hive QA commented on HIVE-21254:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958492/HIVE-21254.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 15788 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=230)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16046/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16046/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16046/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958492 - PreCommit-HIVE-Build
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch,
> HIVE-21254.3.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-21250: --- Resolution: Fixed Status: Resolved (was: Patch Available) Patch Committed to master . Thanks [~harishjp]/[~prasanth_j]! > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-21244: --- Resolution: Duplicate Status: Resolved (was: Patch Available) > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch, HIVE-21244.2.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21259) HiveMetaStoreCilent.getNextNotification throws exception when no new events found
[ https://issues.apache.org/jira/browse/HIVE-21259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek reassigned HIVE-21259: -- > HiveMetaStoreCilent.getNextNotification throws exception when no new events > found > - > > Key: HIVE-21259 > URL: https://issues.apache.org/jira/browse/HIVE-21259 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: anishek >Assignee: anishek >Priority: Major > Fix For: 4.0.0 > > > HiveMetastoreClient can be used to get next notifications for hiveserver2. if > the user has received all notifications and there were no new notifications > generated on the server, then the user should get an empty response, however > currently the client throws an IllegalStateException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21258) Streamline and Add Warning to PrimitiveObjectInspectorFactory.java
[ https://issues.apache.org/jira/browse/HIVE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21258: --- Status: Patch Available (was: Open) > Streamline and Add Warning to PrimitiveObjectInspectorFactory.java > -- > > Key: HIVE-21258 > URL: https://issues.apache.org/jira/browse/HIVE-21258 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-21258.1.patch > > > I just got bit by something pretty good, so I would like to propose adding a > developer-facing warning into the logs to avoid this situation again. Also, > fix up the related cache a bit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21258) Streamline and Add Warning to PrimitiveObjectInspectorFactory.java
[ https://issues.apache.org/jira/browse/HIVE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21258: --- Attachment: HIVE-21258.1.patch > Streamline and Add Warning to PrimitiveObjectInspectorFactory.java > -- > > Key: HIVE-21258 > URL: https://issues.apache.org/jira/browse/HIVE-21258 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-21258.1.patch > > > I just got bit by something pretty good, so I would like to propose adding a > developer-facing warning into the logs to avoid this situation again. Also, > fix up the related cache a bit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21258) Streamline and Add Warning to PrimitiveObjectInspectorFactory.java
[ https://issues.apache.org/jira/browse/HIVE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR reassigned HIVE-21258: -- Assignee: BELUGA BEHR > Streamline and Add Warning to PrimitiveObjectInspectorFactory.java > -- > > Key: HIVE-21258 > URL: https://issues.apache.org/jira/browse/HIVE-21258 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > > I just got bit by something pretty good, so I would like to propose adding a > developer-facing warning into the logs to avoid this situation again. Also, > fix up the related cache a bit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766808#comment-16766808
]
Hive QA commented on HIVE-21254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 9 new +
270 unchanged - 5 fixed = 279 total (was 275) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16046/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16046/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16046/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch,
> HIVE-21254.3.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21257) Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in Hive 3+
[ https://issues.apache.org/jira/browse/HIVE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi reassigned HIVE-21257: - > Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in > Hive 3+ > -- > > Key: HIVE-21257 > URL: https://issues.apache.org/jira/browse/HIVE-21257 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1, 4.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > > After HIVE-19951 is fixed, there still are some cases that vectorized length > UDF doesn't respect the max length of VARCHAR or CHAR. StringLength has an > internal bug. Moreover, it's hard to get input data type details in Hive 2, > unlike Hive 3. So separate both implementation to keep code clean in Hive 3 > while the changes minimal in Hive 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21256) Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in Hive 2
[ https://issues.apache.org/jira/browse/HIVE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi reassigned HIVE-21256: - > Vectorized LENGTH UDF doesn't respect the max length of VARCHAR or CHAR in > Hive 2 > - > > Key: HIVE-21256 > URL: https://issues.apache.org/jira/browse/HIVE-21256 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.4 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > > After HIVE-19951 is fixed, there still are some cases that vectorized length > UDF doesn't respect the max length of VARCHAR or CHAR. StringLength has an > internal bug. Moreover, it's hard to get input data type details in Hive 2, > unlike Hive 3. So separate both implementation to keep code clean in Hive 3 > while the changes minimal in Hive 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766790#comment-16766790
]
Hive QA commented on HIVE-21246:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958486/HIVE-21246.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16045/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16045/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16045/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958486 - PreCommit-HIVE-Build
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch,
> HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766774#comment-16766774
]
Hive QA commented on HIVE-21246:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 506 unchanged - 1
fixed = 506 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16045/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16045/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch,
> HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766763#comment-16766763
]
Hive QA commented on HIVE-21255:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958482/HIVE-21255.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15778 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16044/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16044/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16044/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958482 - PreCommit-HIVE-Build
> Remove QueryConditionBuilder in JdbcStorageHandler
> --
>
> Key: HIVE-21255
> URL: https://issues.apache.org/jira/browse/HIVE-21255
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21255.1.patch
>
>
> QueryConditionBuilder is not correctly implemented. Always see the following
> exception even the query finish succeeded:
> {code}
> 2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|:
> Error during condition build
> java.lang.ArrayIndexOutOfBoundsException: 0
> at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
> at
> org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.2.patch > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch, > HIVE-21240.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766733#comment-16766733
]
Hive QA commented on HIVE-21255:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} jdbc-handler: The patch generated 0 new + 6
unchanged - 15 fixed = 6 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16044/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: jdbc-handler U: jdbc-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16044/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove QueryConditionBuilder in JdbcStorageHandler
> --
>
> Key: HIVE-21255
> URL: https://issues.apache.org/jira/browse/HIVE-21255
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21255.1.patch
>
>
> QueryConditionBuilder is not correctly implemented. Always see the following
> exception even the query finish succeeded:
> {code}
> 2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|:
> Error during condition build
> java.lang.ArrayIndexOutOfBoundsException: 0
> at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
> at
> org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> at
>
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766722#comment-16766722
]
Hive QA commented on HIVE-21254:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958477/HIVE-21254.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15788 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.upgrade.acid.TestPreUpgradeTool.testUpgradeExternalTable
(batchId=335)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16043/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16043/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16043/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958477 - PreCommit-HIVE-Build
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch,
> HIVE-21254.3.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1678#comment-1678 ] Prasanth Jayachandran commented on HIVE-21254: -- Added fix for test cleanup issue. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.3.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch, > HIVE-21254.3.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1675#comment-1675
]
Hive QA commented on HIVE-21254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 9 new +
270 unchanged - 5 fixed = 279 total (was 275) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16043/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16043/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16043/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Status: Patch Available (was: Open)
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch,
> HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Attachment: HIVE-21246.1.patch
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch,
> HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Status: Open (was: Patch Available)
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch,
> HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766652#comment-16766652
]
Hive QA commented on HIVE-21254:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958469/HIVE-21254.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15788 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16042/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16042/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16042/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958469 - PreCommit-HIVE-Build
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-21255:
--
Attachment: HIVE-21255.1.patch
> Remove QueryConditionBuilder in JdbcStorageHandler
> --
>
> Key: HIVE-21255
> URL: https://issues.apache.org/jira/browse/HIVE-21255
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21255.1.patch
>
>
> QueryConditionBuilder is not correctly implemented. Always see the following
> exception even the query finish succeeded:
> {code}
> 2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|:
> Error during condition build
> java.lang.ArrayIndexOutOfBoundsException: 0
> at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
> at
> org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> We don't actually need QueryConditionBuilder when cbo is enabled since
> predicate push down is handled by calcite (HIVE-20822). One can argue when
> cbo is disabled we might still need that since calcite will not do the push
> down, but that's a minor code path and removing QueryConditionBuilder won't
> cause any correctness issue. So I'd like to remove it for simplicity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-21255:
--
Status: Patch Available (was: Open)
> Remove QueryConditionBuilder in JdbcStorageHandler
> --
>
> Key: HIVE-21255
> URL: https://issues.apache.org/jira/browse/HIVE-21255
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21255.1.patch
>
>
> QueryConditionBuilder is not correctly implemented. Always see the following
> exception even the query finish succeeded:
> {code}
> 2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|:
> Error during condition build
> java.lang.ArrayIndexOutOfBoundsException: 0
> at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
> at
> org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> We don't actually need QueryConditionBuilder when cbo is enabled since
> predicate push down is handled by calcite (HIVE-20822). One can argue when
> cbo is disabled we might still need that since calcite will not do the push
> down, but that's a minor code path and removing QueryConditionBuilder won't
> cause any correctness issue. So I'd like to remove it for simplicity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21255) Remove QueryConditionBuilder in JdbcStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai reassigned HIVE-21255:
-
> Remove QueryConditionBuilder in JdbcStorageHandler
> --
>
> Key: HIVE-21255
> URL: https://issues.apache.org/jira/browse/HIVE-21255
> Project: Hive
> Issue Type: Improvement
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
>
> QueryConditionBuilder is not correctly implemented. Always see the following
> exception even the query finish succeeded:
> {code}
> 2019-02-13 01:09:53,406 [ERROR] [TezChild] |jdbc.QueryConditionBuilder|:
> Error during condition build
> java.lang.ArrayIndexOutOfBoundsException: 0
> at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.createConditionString(QueryConditionBuilder.java:125)
> at
> org.apache.hive.storage.jdbc.QueryConditionBuilder.buildCondition(QueryConditionBuilder.java:74)
> at
> org.apache.hive.storage.jdbc.conf.JdbcStorageConfigManager.getQueryToExecute(JdbcStorageConfigManager.java:155)
> at
> org.apache.hive.storage.jdbc.dao.GenericJdbcDatabaseAccessor.getRecordIterator(GenericJdbcDatabaseAccessor.java:158)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:58)
> at
> org.apache.hive.storage.jdbc.JdbcRecordReader.next(JdbcRecordReader.java:35)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> We don't actually need QueryConditionBuilder when cbo is enabled since
> predicate push down is handled by calcite (HIVE-20822). One can argue when
> cbo is disabled we might still need that since calcite will not do the push
> down, but that's a minor code path and removing QueryConditionBuilder won't
> cause any correctness issue. So I'd like to remove it for simplicity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21242) Calcite Planner Logging Indicates UTF-16 Encoding
[
https://issues.apache.org/jira/browse/HIVE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766635#comment-16766635
]
Gopal V commented on HIVE-21242:
Java used to use UCS-2, it switched to UTF-16 by default to support
supplemental characters
https://docs.oracle.com/javase/8/docs/technotes/guides/intl/overview.html#textrep
{code}
The primitive data type char in the Java programming language is an unsigned
16-bit integer that can represent a Unicode code point in the range U+ to
U+, or the code units of UTF-16.
{code}
> Calcite Planner Logging Indicates UTF-16 Encoding
> -
>
> Key: HIVE-21242
> URL: https://issues.apache.org/jira/browse/HIVE-21242
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Priority: Major
>
> I noticed some debug logging from calcite and it is using UTF-16. I would
> expect UTF-8.
> {code}
> 2019-02-10T19:08:06,393 DEBUG [7db4d3c5-0f88-49db-88fa-ad6428c23784 main]
> parse.CalcitePlanner: Plan after decorrelation:
> HiveSortLimit(offset=[0], fetch=[2])
> HiveProject(_o__c0=[array(3, 2, 1)], _o__c1=[map(1, 2001-01-01, 2, null)],
> _o__c2=[named_struct(_UTF-16LE'c1', 123456, _UTF-16LE'c2', _UTF-16LE'hello',
> _UTF-16LE'c3', array(_UTF-16LE'aa', _UTF-16LE'bb', _UTF-16LE'cc'),
> _UTF-16LE'c4', map(_UTF-16LE'abc', 123, _UTF-16LE'xyz', 456), _UTF-16LE'c5',
> named_struct(_UTF-16LE'c5_1', _UTF-16LE'bye', _UTF-16LE'c5_2', 88))])
> HiveTableScan(table=[[default, src]], table:alias=[src])
> {code}
> I'm not sure if this is a calcite internal thing which can be configured or
> if this only an artifact of the way the logging works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766638#comment-16766638 ] Jason Dere commented on HIVE-21254: --- +1 > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.2.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766630#comment-16766630 ] Prasanth Jayachandran commented on HIVE-21254: -- Handled ACL exception with a warning to run manual compaction per offline discussion with [~jdere] > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch, HIVE-21254.2.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[
https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766614#comment-16766614
]
Hive QA commented on HIVE-21254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 9 new +
270 unchanged - 5 fixed = 279 total (was 275) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16042/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16042/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16042/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Pre-upgrade tool should handle exceptions and skip db/tables
>
>
> Key: HIVE-21254
> URL: https://issues.apache.org/jira/browse/HIVE-21254
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21254.1.patch
>
>
> When exceptions like AccessControlException is thrown, pre-upgrade tool
> fails. If hive user does not have read access to database or tables (some
> external tables denies read access to hive), pre-upgrade tool should just
> assume they are external tables and move on without failing pre-upgrade
> process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21253) Support DB2 in JDBC StorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766605#comment-16766605
]
Hive QA commented on HIVE-21253:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958453/HIVE-21253.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16041/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16041/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16041/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958453 - PreCommit-HIVE-Build
> Support DB2 in JDBC StorageHandler
> --
>
> Key: HIVE-21253
> URL: https://issues.apache.org/jira/browse/HIVE-21253
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21253.1.patch
>
>
> Make DB2 a first class member of JdbcStorageHandler. It can even work before
> the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This
> patch make it a standard feature. Note this is only for DB2 tables as
> external JdbcStorageHandler table. We haven't tested DB2 as a metastore
> backend and it's not a goal for this ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766602#comment-16766602 ] Prasanth Jayachandran commented on HIVE-21254: -- Make sense. Will update patch to handle only AccessControlException. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766599#comment-16766599 ] Ashutosh Chauhan commented on HIVE-21254: - It should be determined whether table is acid or not. Pre-upgrade tool need to compact acid tables. Thats a must, we can't skip it. If it's a non-acid table then no action is needed on it anyways, so ACE exception can be ignored, but if its an acid table then I think tool shall raise an exception and fail. User need to make sure that user running pre-upgrade tool has sufficient privileges. > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766595#comment-16766595 ] Prasanth Jayachandran commented on HIVE-21254: -- Attaching RB link. [~ekoifman]/ [~jdere] can someone please review this patch? > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Status: Patch Available (was: Open) > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21254: - Attachment: HIVE-21254.1.patch > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21254.1.patch > > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21254) Pre-upgrade tool should handle exceptions and skip db/tables
[ https://issues.apache.org/jira/browse/HIVE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-21254: > Pre-upgrade tool should handle exceptions and skip db/tables > > > Key: HIVE-21254 > URL: https://issues.apache.org/jira/browse/HIVE-21254 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > When exceptions like AccessControlException is thrown, pre-upgrade tool > fails. If hive user does not have read access to database or tables (some > external tables denies read access to hive), pre-upgrade tool should just > assume they are external tables and move on without failing pre-upgrade > process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21253) Support DB2 in JDBC StorageHandler
[
https://issues.apache.org/jira/browse/HIVE-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766576#comment-16766576
]
Hive QA commented on HIVE-21253:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16041/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: jdbc-handler U: jdbc-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16041/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support DB2 in JDBC StorageHandler
> --
>
> Key: HIVE-21253
> URL: https://issues.apache.org/jira/browse/HIVE-21253
> Project: Hive
> Issue Type: Improvement
> Components: StorageHandler
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21253.1.patch
>
>
> Make DB2 a first class member of JdbcStorageHandler. It can even work before
> the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This
> patch make it a standard feature. Note this is only for DB2 tables as
> external JdbcStorageHandler table. We haven't tested DB2 as a metastore
> backend and it's not a goal for this ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21252) LazyTimestamp - Use String Equals
[
https://issues.apache.org/jira/browse/HIVE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766565#comment-16766565
]
Hive QA commented on HIVE-21252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958443/HIVE-21252.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16040/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16040/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16040/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958443 - PreCommit-HIVE-Build
> LazyTimestamp - Use String Equals
> -
>
> Key: HIVE-21252
> URL: https://issues.apache.org/jira/browse/HIVE-21252
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21252.1.patch
>
>
> {code:java|title=LazyTimestamp.java}
> if (s.compareTo("NULL") == 0) {
> isNull = true;
> logExceptionMessage(bytes, start, length, "TIMESTAMP");
> }
> {code}
> compareTo generates a number to represent the differences between the two
> Strings. It's faster to simply call "equals" which will simply compare the
> two String directly and return a boolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21252) LazyTimestamp - Use String Equals
[
https://issues.apache.org/jira/browse/HIVE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766532#comment-16766532
]
Hive QA commented on HIVE-21252:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16040/dev-support/hive-personality.sh
|
| git revision | master / 793f192 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16040/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LazyTimestamp - Use String Equals
> -
>
> Key: HIVE-21252
> URL: https://issues.apache.org/jira/browse/HIVE-21252
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21252.1.patch
>
>
> {code:java|title=LazyTimestamp.java}
> if (s.compareTo("NULL") == 0) {
> isNull = true;
> logExceptionMessage(bytes, start, length, "TIMESTAMP");
> }
> {code}
> compareTo generates a number to represent the differences between the two
> Strings. It's faster to simply call "equals" which will simply compare the
> two String directly and return a boolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21253) Support DB2 in JDBC StorageHandler
[ https://issues.apache.org/jira/browse/HIVE-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21253: -- Attachment: HIVE-21253.1.patch > Support DB2 in JDBC StorageHandler > -- > > Key: HIVE-21253 > URL: https://issues.apache.org/jira/browse/HIVE-21253 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21253.1.patch > > > Make DB2 a first class member of JdbcStorageHandler. It can even work before > the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This > patch make it a standard feature. Note this is only for DB2 tables as > external JdbcStorageHandler table. We haven't tested DB2 as a metastore > backend and it's not a goal for this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21253) Support DB2 in JDBC StorageHandler
[ https://issues.apache.org/jira/browse/HIVE-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21253: -- Status: Patch Available (was: Open) > Support DB2 in JDBC StorageHandler > -- > > Key: HIVE-21253 > URL: https://issues.apache.org/jira/browse/HIVE-21253 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21253.1.patch > > > Make DB2 a first class member of JdbcStorageHandler. It can even work before > the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This > patch make it a standard feature. Note this is only for DB2 tables as > external JdbcStorageHandler table. We haven't tested DB2 as a metastore > backend and it's not a goal for this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21253) Support DB2 in JDBC StorageHandler
[ https://issues.apache.org/jira/browse/HIVE-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-21253: - > Support DB2 in JDBC StorageHandler > -- > > Key: HIVE-21253 > URL: https://issues.apache.org/jira/browse/HIVE-21253 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > > Make DB2 a first class member of JdbcStorageHandler. It can even work before > the patch by using POSTGRES as DB type and add db2 jdbc jar manually. This > patch make it a standard feature. Note this is only for DB2 tables as > external JdbcStorageHandler table. We haven't tested DB2 as a metastore > backend and it's not a goal for this ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[
https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766518#comment-16766518
]
Hive QA commented on HIVE-21217:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958422/HIVE-21217.0.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15796 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_range_multiorder]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_range_multiorder]
(batchId=163)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16039/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16039/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16039/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958422 - PreCommit-HIVE-Build
> Optimize range calculation for PTF
> --
>
> Key: HIVE-21217
> URL: https://issues.apache.org/jira/browse/HIVE-21217
> Project: Hive
> Issue Type: Improvement
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-21217.0.patch
>
>
> During window function execution Hive has to iterate on neighbouring rows of
> the current row to find the beginning and end of the proper range (on which
> the aggregation will be executed).
> When we're using range based windows and have many rows with a certain key
> value this can take a lot of time. (e.g. partition size of 80M, in which we
> have 2 ranges of 40M rows according to the orderby column: within these 40M
> rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity)
> I propose to introduce a cache that keeps track of already calculated range
> ends so it can be reused in future scans.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[
https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766464#comment-16766464
]
Hive QA commented on HIVE-21217:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 12 new + 139 unchanged - 34
fixed = 151 total (was 173) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
6s{color} | {color:red} ql generated 6 new + 2295 unchanged - 0 fixed = 2301
total (was 2295) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
56s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Suspicious comparison of Integer references in
org.apache.hadoop.hive.ql.exec.BoundaryCache.put(Integer, Object) At
BoundaryCache.java:in org.apache.hadoop.hive.ql.exec.BoundaryCache.put(Integer,
Object) At BoundaryCache.java:[line 36] |
| | Load of known null value in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeEndCurrentRow(int,
PTFPartition) At ValueBoundaryScanner.java:in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeEndCurrentRow(int,
PTFPartition) At ValueBoundaryScanner.java:[line 531] |
| | Load of known null value in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeEndFollowing(int,
PTFPartition) At ValueBoundaryScanner.java:in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeEndFollowing(int,
PTFPartition) At ValueBoundaryScanner.java:[line 569] |
| | Load of known null value in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeStartCurrentRow(int,
PTFPartition) At ValueBoundaryScanner.java:in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeStartCurrentRow(int,
PTFPartition) At ValueBoundaryScanner.java:[line 381] |
| | Load of known null value in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeStartFollowing(int,
PTFPartition) At ValueBoundaryScanner.java:in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeStartFollowing(int,
PTFPartition) At ValueBoundaryScanner.java:[line 414] |
| | Load of known null value in
org.apache.hadoop.hive.ql.udf.ptf.SingleValueBoundaryScanner.computeStartPreceding(int,
PTFPartition) At ValueBoundaryScanner.java:in
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766420#comment-16766420
]
Hive QA commented on HIVE-21240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958395/HIVE-21240.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 15791 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_json_read]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
(batchId=239)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFJsonRead.testList
(batchId=293)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFJsonRead.testListNull
(batchId=293)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFJsonRead.testMap
(batchId=293)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFJsonRead.testSimpleStruct
(batchId=293)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFJsonRead.testStructNullComplexField
(batchId=293)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFJsonRead.testStructNullField
(batchId=293)
org.apache.hive.hcatalog.data.TestJsonSerDe.testLooseJsonReadability
(batchId=215)
org.apache.hive.hcatalog.data.TestJsonSerDe.testRW (batchId=215)
org.apache.hive.hcatalog.pig.TestHCatStorer.testDateCharTypes[6] (batchId=206)
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteChar[6] (batchId=206)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testJsonInputStreamDP
(batchId=334)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16038/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16038/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16038/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958395 - PreCommit-HIVE-Build
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.1.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JSON SerDe has a few issues, I will link them to this JIRA.
> * Use Jackson Tree parser instead of manually parsing
> * Added support for base-64 encoded data (the expected format when using JSON)
> * Added support to skip blank lines (returns all columns as null values)
> * Current JSON parser accepts, but does not apply, custom timestamp formats
> in most cases
> * Added some unit tests
> * Added cache for column-name to column-index searches, currently O\(n\) for
> each row processed, for each column in the row
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21240) JSON SerDe Deserialize Re-Write
[
https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766397#comment-16766397
]
Hive QA commented on HIVE-21240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} serde: The patch generated 8 new + 20 unchanged - 9
fixed = 28 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red} serde generated 2 new + 193 unchanged - 4 fixed = 195
total (was 197) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.getColIndex(String) At
HiveJsonStructReader.java:org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.getColIndex(String)
At HiveJsonStructReader.java:[line 456] |
| | Found reliance on default encoding in
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.getByteValue(JsonNode):in
org.apache.hadoop.hive.serde2.json.HiveJsonStructReader.getByteValue(JsonNode):
String.getBytes() At HiveJsonStructReader.java:[line 375] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16038/dev-support/hive-personality.sh
|
| git revision | master / 598cb81 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16038/yetus/diff-checkstyle-serde.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16038/yetus/new-findbugs-serde.html
|
| modules | C: serde ql hcatalog/core U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16038/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JSON SerDe Deserialize Re-Write
> ---
>
> Key: HIVE-21240
> URL: https://issues.apache.org/jira/browse/HIVE-21240
>
[jira] [Updated] (HIVE-21252) LazyTimestamp - Use String Equals
[
https://issues.apache.org/jira/browse/HIVE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21252:
---
Status: Patch Available (was: Open)
Please consider for 3.x and 4.0 branches.
> LazyTimestamp - Use String Equals
> -
>
> Key: HIVE-21252
> URL: https://issues.apache.org/jira/browse/HIVE-21252
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21252.1.patch
>
>
> {code:java|title=LazyTimestamp.java}
> if (s.compareTo("NULL") == 0) {
> isNull = true;
> logExceptionMessage(bytes, start, length, "TIMESTAMP");
> }
> {code}
> compareTo generates a number to represent the differences between the two
> Strings. It's faster to simply call "equals" which will simply compare the
> two String directly and return a boolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21252) LazyTimestamp - Use String Equals
[
https://issues.apache.org/jira/browse/HIVE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-21252:
--
Assignee: BELUGA BEHR
> LazyTimestamp - Use String Equals
> -
>
> Key: HIVE-21252
> URL: https://issues.apache.org/jira/browse/HIVE-21252
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21252.1.patch
>
>
> {code:java|title=LazyTimestamp.java}
> if (s.compareTo("NULL") == 0) {
> isNull = true;
> logExceptionMessage(bytes, start, length, "TIMESTAMP");
> }
> {code}
> compareTo generates a number to represent the differences between the two
> Strings. It's faster to simply call "equals" which will simply compare the
> two String directly and return a boolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21252) LazyTimestamp - Use String Equals
[
https://issues.apache.org/jira/browse/HIVE-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21252:
---
Attachment: HIVE-21252.1.patch
> LazyTimestamp - Use String Equals
> -
>
> Key: HIVE-21252
> URL: https://issues.apache.org/jira/browse/HIVE-21252
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21252.1.patch
>
>
> {code:java|title=LazyTimestamp.java}
> if (s.compareTo("NULL") == 0) {
> isNull = true;
> logExceptionMessage(bytes, start, length, "TIMESTAMP");
> }
> {code}
> compareTo generates a number to represent the differences between the two
> Strings. It's faster to simply call "equals" which will simply compare the
> two String directly and return a boolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766364#comment-16766364 ] Vineet Garg commented on HIVE-21238: Pushed to master. Thanks [~mgergely] > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21238: --- Resolution: Fixed Fix Version/s: (was: 3.0.1) 4.0.0 Status: Resolved (was: Patch Available) > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21238) Fix Surrogate Key issue
[ https://issues.apache.org/jira/browse/HIVE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766361#comment-16766361 ] Vineet Garg commented on HIVE-21238: +1. > Fix Surrogate Key issue > --- > > Key: HIVE-21238 > URL: https://issues.apache.org/jira/browse/HIVE-21238 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-21238.01.patch, HIVE-21238.02.patch > > > When SemanticAnalyzer is trying to set the write id for the surrogate key > function it is looking for such functions only in the main operator, while it > should also look for it in it's parents. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766352#comment-16766352
]
Hive QA commented on HIVE-21246:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958394/HIVE-21246.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15783 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=86)
org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithHttp.org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithHttp
(batchId=259)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16037/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16037/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16037/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958394 - PreCommit-HIVE-Build
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21249) Reduce memory footprint in ObjectStore.refreshPrivileges
[
https://issues.apache.org/jira/browse/HIVE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766313#comment-16766313
]
Daniel Dai commented on HIVE-21249:
---
[~sankarh], can you review it?
> Reduce memory footprint in ObjectStore.refreshPrivileges
> -
>
> Key: HIVE-21249
> URL: https://issues.apache.org/jira/browse/HIVE-21249
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21249.1.patch
>
>
> We found there're could be many records in TBL_COL_PRIVS for a single table
> (a table granted to many users), thus result a OOM in
> ObjectStore.listTableAllColumnGrants. We shall reduce the memory footprint
> for ObjectStore.refreshPrivileges. Here is the stack of OOM:
> {code}
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:690)
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:710)
> org.apache.hadoop.hive.metastore.ObjectStore.listTableAllColumnGrants(ObjectStore.java:6629)
> org.apache.hadoop.hive.metastore.ObjectStore.refreshPrivileges(ObjectStore.java:6200)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> com.sun.proxy.$Proxy32.refreshPrivileges(, line not available)
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.refresh_privileges(HiveMetaStore.java:6507)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> com.sun.proxy.$Proxy34.refresh_privileges(, line not available)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17608)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$refresh_privileges.getResult(ThriftHiveMetastore.java:17592)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:636)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:631)
> java.security.AccessController.doPrivileged(Native method)
> javax.security.auth.Subject.doAs(Subject.java:422)
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:631)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766316#comment-16766316
]
Hive QA commented on HIVE-21246:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 506 unchanged - 1
fixed = 506 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16037/dev-support/hive-personality.sh
|
| git revision | master / 598cb81 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16037/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[
https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766293#comment-16766293
]
Hive QA commented on HIVE-21250:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958347/HIVE-21250.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16036/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16036/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16036/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958347 - PreCommit-HIVE-Build
> NPE in HiveProtoLoggingHook for eventPerFile mode.
> --
>
> Key: HIVE-21250
> URL: https://issues.apache.org/jira/browse/HIVE-21250
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Critical
> Attachments: HIVE-21250.01.patch
>
>
> When eventPerFile is enabled, writer is set to null after the first event, it
> causes an NPE in the next path until handleTick comes back.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21236) SharedWorkOptimizer should check table properties
[ https://issues.apache.org/jira/browse/HIVE-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21236: --- Resolution: Fixed Fix Version/s: 3.1.2 3.2.0 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, branch-3, branch-3.1. Thanks [~ashutoshc] > SharedWorkOptimizer should check table properties > - > > Key: HIVE-21236 > URL: https://issues.apache.org/jira/browse/HIVE-21236 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0, 3.2.0, 3.1.2 > > Attachments: HIVE-21236.01.patch, HIVE-21236.02.patch, > HIVE-21236.patch > > > For instance, Calcite may have pushed computation to Druid or a JDBC source, > rest of table structures may look the same, but the embedded query is > different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21236) SharedWorkOptimizer should check table properties
[ https://issues.apache.org/jira/browse/HIVE-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766279#comment-16766279 ] Ashutosh Chauhan commented on HIVE-21236: - +1 > SharedWorkOptimizer should check table properties > - > > Key: HIVE-21236 > URL: https://issues.apache.org/jira/browse/HIVE-21236 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21236.01.patch, HIVE-21236.02.patch, > HIVE-21236.patch > > > For instance, Calcite may have pushed computation to Druid or a JDBC source, > rest of table structures may look the same, but the embedded query is > different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21236) SharedWorkOptimizer should check table properties
[ https://issues.apache.org/jira/browse/HIVE-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766268#comment-16766268 ] Jesus Camacho Rodriguez commented on HIVE-21236: [~ashutoshc], could you take a look? Thanks > SharedWorkOptimizer should check table properties > - > > Key: HIVE-21236 > URL: https://issues.apache.org/jira/browse/HIVE-21236 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21236.01.patch, HIVE-21236.02.patch, > HIVE-21236.patch > > > For instance, Calcite may have pushed computation to Druid or a JDBC source, > rest of table structures may look the same, but the embedded query is > different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[
https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766260#comment-16766260
]
Hive QA commented on HIVE-21250:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16036/dev-support/hive-personality.sh
|
| git revision | master / e540cf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16036/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE in HiveProtoLoggingHook for eventPerFile mode.
> --
>
> Key: HIVE-21250
> URL: https://issues.apache.org/jira/browse/HIVE-21250
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Critical
> Attachments: HIVE-21250.01.patch
>
>
> When eventPerFile is enabled, writer is set to null after the first event, it
> causes an NPE in the next path until handleTick comes back.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work stopped] (HIVE-18101) ISO 8601 UDF
[
https://issues.apache.org/jira/browse/HIVE-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-18101 stopped by BELUGA BEHR.
--
> ISO 8601 UDF
>
>
> Key: HIVE-18101
> URL: https://issues.apache.org/jira/browse/HIVE-18101
> Project: Hive
> Issue Type: New Feature
> Components: Contrib
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-18101.1.patch
>
>
> Please consider my submission of a UDF for parsing ISO 8601 strings into
> timestamps. It uses Java 1.8's
> {{java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME}} class.
> https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
> https://en.wikipedia.org/wiki/ISO_8601
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-21217: -- Status: Patch Available (was: In Progress) > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766235#comment-16766235 ] Adam Szita commented on HIVE-21217: --- Attached WIP patch to be checked by ptest > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
[
https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766233#comment-16766233
]
Hive QA commented on HIVE-21218:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958362/HIVE-21218.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15786 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniHiveKafkaCliDriver.testCliDriver[kafka_storage_handler]
(batchId=275)
org.apache.hadoop.hive.ql.TestTxnCommands.testMergeOnTezEdges (batchId=327)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16035/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16035/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16035/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958362 - PreCommit-HIVE-Build
> KafkaSerDe doesn't support topics created via Confluent Avro serializer
> ---
>
> Key: HIVE-21218
> URL: https://issues.apache.org/jira/browse/HIVE-21218
> Project: Hive
> Issue Type: Bug
> Components: kafka integration, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Milan Baran
>Assignee: Milan Baran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21218.2.patch, HIVE-21218.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> According to [Google
> groups|https://groups.google.com/forum/#!topic/confluent-platform/JYhlXN0u9_A]
> the Confluent avro serialzier uses propertiary format for kafka value -
> <4 bytes of schema ID> conforms to schema>.
> This format does not cause any problem for Confluent kafka deserializer which
> respect the format however for hive kafka handler its bit a problem to
> correctly deserialize kafka value, because Hive uses custom deserializer from
> bytes to objects and ignores kafka consumer ser/deser classes provided via
> table property.
> It would be nice to support Confluent format with magic byte.
> Also it would be great to support Schema registry as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21217) Optimize range calculation for PTF
[ https://issues.apache.org/jira/browse/HIVE-21217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-21217: -- Attachment: HIVE-21217.0.patch > Optimize range calculation for PTF > -- > > Key: HIVE-21217 > URL: https://issues.apache.org/jira/browse/HIVE-21217 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Major > Attachments: HIVE-21217.0.patch > > > During window function execution Hive has to iterate on neighbouring rows of > the current row to find the beginning and end of the proper range (on which > the aggregation will be executed). > When we're using range based windows and have many rows with a certain key > value this can take a lot of time. (e.g. partition size of 80M, in which we > have 2 ranges of 40M rows according to the orderby column: within these 40M > rowsets we're doing 40M x 40M/2 steps.. which is of n^2 time complexity) > I propose to introduce a cache that keeps track of already calculated range > ends so it can be reused in future scans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18101) ISO 8601 UDF
[
https://issues.apache.org/jira/browse/HIVE-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR resolved HIVE-18101.
Resolution: Won't Fix
> ISO 8601 UDF
>
>
> Key: HIVE-18101
> URL: https://issues.apache.org/jira/browse/HIVE-18101
> Project: Hive
> Issue Type: New Feature
> Components: Contrib
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-18101.1.patch
>
>
> Please consider my submission of a UDF for parsing ISO 8601 strings into
> timestamps. It uses Java 1.8's
> {{java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME}} class.
> https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
> https://en.wikipedia.org/wiki/ISO_8601
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-18101) ISO 8601 UDF
[
https://issues.apache.org/jira/browse/HIVE-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-18101 started by BELUGA BEHR.
--
> ISO 8601 UDF
>
>
> Key: HIVE-18101
> URL: https://issues.apache.org/jira/browse/HIVE-18101
> Project: Hive
> Issue Type: New Feature
> Components: Contrib
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-18101.1.patch
>
>
> Please consider my submission of a UDF for parsing ISO 8601 strings into
> timestamps. It uses Java 1.8's
> {{java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME}} class.
> https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
> https://en.wikipedia.org/wiki/ISO_8601
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18101) ISO 8601 UDF
[
https://issues.apache.org/jira/browse/HIVE-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18101:
---
Status: Open (was: Patch Available)
> ISO 8601 UDF
>
>
> Key: HIVE-18101
> URL: https://issues.apache.org/jira/browse/HIVE-18101
> Project: Hive
> Issue Type: New Feature
> Components: Contrib
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-18101.1.patch
>
>
> Please consider my submission of a UDF for parsing ISO 8601 strings into
> timestamps. It uses Java 1.8's
> {{java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME}} class.
> https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
> https://en.wikipedia.org/wiki/ISO_8601
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Attachment: HIVE-12679.branch-2.3.patch > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Reporter: Austin Lee >Assignee: Rentao Wu >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.1.patch, HIVE-12679.2.patch, > HIVE-12679.branch-1.2.patch, HIVE-12679.branch-2.3.patch, HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-12679) Allow users to be able to specify an implementation of IMetaStoreClient via HiveConf
[ https://issues.apache.org/jira/browse/HIVE-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Lee updated HIVE-12679: -- Attachment: HIVE-12679.branch-1.2.patch > Allow users to be able to specify an implementation of IMetaStoreClient via > HiveConf > > > Key: HIVE-12679 > URL: https://issues.apache.org/jira/browse/HIVE-12679 > Project: Hive > Issue Type: Improvement > Components: Configuration, Metastore, Query Planning >Reporter: Austin Lee >Assignee: Rentao Wu >Priority: Minor > Labels: metastore > Attachments: HIVE-12679.1.patch, HIVE-12679.2.patch, > HIVE-12679.branch-1.2.patch, HIVE-12679.branch-2.3.patch, HIVE-12679.patch > > > Hi, > I would like to propose a change that would make it possible for users to > choose an implementation of IMetaStoreClient via HiveConf, i.e. > hive-site.xml. Currently, in Hive the choice is hard coded to be > SessionHiveMetaStoreClient in org.apache.hadoop.hive.ql.metadata.Hive. There > is no other direct reference to SessionHiveMetaStoreClient other than the > hard coded class name in Hive.java and the QL component operates only on the > IMetaStoreClient interface so the change would be minimal and it would be > quite similar to how an implementation of RawStore is specified and loaded in > hive-metastore. One use case this change would serve would be one where a > user wishes to use an implementation of this interface without the dependency > on the Thrift server. > > Thank you, > Austin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21218) KafkaSerDe doesn't support topics created via Confluent Avro serializer
[
https://issues.apache.org/jira/browse/HIVE-21218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766182#comment-16766182
]
Hive QA commented on HIVE-21218:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} kafka-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} kafka-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} kafka-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} kafka-handler: The patch generated 73 new + 1
unchanged - 0 fixed = 74 total (was 1) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} kafka-handler in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16035/dev-support/hive-personality.sh
|
| git revision | master / e540cf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/patch-mvninstall-kafka-handler.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/patch-compile-kafka-handler.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/patch-compile-kafka-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/diff-checkstyle-kafka-handler.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/patch-findbugs-kafka-handler.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus/patch-asflicense-problems.txt
|
| modules | C: serde kafka-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16035/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> KafkaSerDe doesn't support topics created via Confluent Avro serializer
> ---
>
> Key: HIVE-21218
> URL: https://issues.apache.org/jira/browse/HIVE-21218
> Project: Hive
> Issue Type: Bug
>
[jira] [Commented] (HIVE-21232) LLAP: Add a cache-miss friendly split affinity provider
[
https://issues.apache.org/jira/browse/HIVE-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766153#comment-16766153
]
Hive QA commented on HIVE-21232:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958360/HIVE-21232.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15787 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority]
(batchId=191)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16034/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16034/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16034/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958360 - PreCommit-HIVE-Build
> LLAP: Add a cache-miss friendly split affinity provider
> ---
>
> Key: HIVE-21232
> URL: https://issues.apache.org/jira/browse/HIVE-21232
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-21232.1.patch
>
>
> If one of the LLAP nodes have data-locality, preferring that over another
> does have advantages for the first query or a more general cache-miss.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21198) Introduce a database object reference class
[
https://issues.apache.org/jira/browse/HIVE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Lavati reassigned HIVE-21198:
---
Assignee: David Lavati
> Introduce a database object reference class
> ---
>
> Key: HIVE-21198
> URL: https://issues.apache.org/jira/browse/HIVE-21198
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: David Lavati
>Priority: Major
>
> There are many places in which "{databasename}.{tablename}" is passed as a
> single string; there are some places where the they travel as 2 separate
> arguments.
> Idea would be to introduce a simple immutable class with 2 fields ; and pass
> these informations together. Making this better is required if we would be
> wanting to enable dot in tablenames
> HIVE-16907, HIVE-21151
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21232) LLAP: Add a cache-miss friendly split affinity provider
[
https://issues.apache.org/jira/browse/HIVE-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766096#comment-16766096
]
Hive QA commented on HIVE-21232:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 7 new + 30 unchanged - 0 fixed
= 37 total (was 30) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 1 new + 2295 unchanged - 0 fixed = 2296
total (was 2295) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Null passed for non-null parameter of new java.util.HashSet(Collection)
in new
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider(List)
Method invoked at HostAffinitySplitLocationProvider.java:of new
java.util.HashSet(Collection) in new
org.apache.hadoop.hive.ql.exec.tez.HostAffinitySplitLocationProvider(List)
Method invoked at HostAffinitySplitLocationProvider.java:[line 63] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16034/dev-support/hive-personality.sh
|
| git revision | master / e540cf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16034/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16034/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16034/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16034/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Add a cache-miss friendly split affinity provider
> ---
>
> Key: HIVE-21232
> URL: https://issues.apache.org/jira/browse/HIVE-21232
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-21232.1.patch
>
>
> If one of the LLAP nodes have data-locality, preferring that over another
> does have advantages for the first query or a more general cache-miss.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21055) REPL LOAD command executing copy in serial mode even if parallel execution is enabled using with clause
[ https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21055: Summary: REPL LOAD command executing copy in serial mode even if parallel execution is enabled using with clause (was: Replication load command executing copy in serial mode even if parallel execution is enabled using with clause) > REPL LOAD command executing copy in serial mode even if parallel execution is > enabled using with clause > --- > > Key: HIVE-21055 > URL: https://issues.apache.org/jira/browse/HIVE-21055 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch, > HIVE-21055.03.patch > > > For repl load command use can specify the execution mode as part of "with" > clause. But the config for executing task in parallel or serial is not read > from the command specific config. It is read from the hive server config. So > even if user specifies to run the tasks in parallel during repl load command, > the tasks are getting executed serially. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-21144) ODBC with prepared statement fail inside a CTE
[
https://issues.apache.org/jira/browse/HIVE-21144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guillaume resolved HIVE-21144.
--
Resolution: Won't Fix
Bug outside Hive, in Simba ODBC driver.
> ODBC with prepared statement fail inside a CTE
> --
>
> Key: HIVE-21144
> URL: https://issues.apache.org/jira/browse/HIVE-21144
> Project: Hive
> Issue Type: Bug
> Components: ODBC
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> I am trying to execute a very simple query, using python/pyodbc on Windows
> (with a working system-wide odbc DSN: HiveProd):
> {code:java}
> import pyodbc cnxn = pyodbc.connect('DSN=HiveProd', autocommit=True)
> cursor = cnxn.cursor()
> # works
> q="select ? as lic, ? as cpg"
> # fails
> q="with init as (select ? as lic, ? as cpg) select * from init"
> cursor.execute(q, '1', 'some string')
> for row in cursor:
> print(row.lic, row.cpg)
> {code}
> Basically, create an odbc connection, run a query with a prepared statement
> and print the result.
> A basic query works fine. If I put this query inside a CTE, I get:
> {{ cursor.execute("with init as (select ? as lic, ? as cpg) select * from
> init", '1', 'some string') pyodbc.ProgrammingError: ('42000', "[42000]
> [Hortonworks][Hardy] (80) Syntax or semantic analysis error thrown in server
> while executing query. Error message from server: Error while compiling
> statement: FAILED: ParseException line 1:21 can not recognize input near '?'
> 'as' 'lic' in select clause (80) (SQLPrepare)")}}
> This is not specific to python as I get the same issue with .Net.
> Trying the same with JDBC works fine.
> Testing on Hive 3.1.0 from Hdp 3.1.0
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[
https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766064#comment-16766064
]
Hive QA commented on HIVE-21244:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958356/HIVE-21244.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15787 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=243)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16033/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16033/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16033/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958356 - PreCommit-HIVE-Build
> NPE in Hive Proto Logger
>
>
> Key: HIVE-21244
> URL: https://issues.apache.org/jira/browse/HIVE-21244
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21244.1.patch, HIVE-21244.2.patch
>
>
> [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308]
> can cause NPE. There is no uncaught exception handler for this thread. This
> NPE can silently fail and drop the event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21055) REPL LOAD command executing copy in serial mode even if parallel execution is enabled using WITH clause
[ https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766059#comment-16766059 ] Sankar Hariappan commented on HIVE-21055: - +1 > REPL LOAD command executing copy in serial mode even if parallel execution is > enabled using WITH clause > --- > > Key: HIVE-21055 > URL: https://issues.apache.org/jira/browse/HIVE-21055 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch, > HIVE-21055.03.patch > > > For repl load command use can specify the execution mode as part of "with" > clause. But the config for executing task in parallel or serial is not read > from the command specific config. It is read from the hive server config. So > even if user specifies to run the tasks in parallel during repl load command, > the tasks are getting executed serially. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21055) REPL LOAD command executing copy in serial mode even if parallel execution is enabled using WITH clause
[ https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21055: Summary: REPL LOAD command executing copy in serial mode even if parallel execution is enabled using WITH clause (was: REPL LOAD command executing copy in serial mode even if parallel execution is enabled using with clause) > REPL LOAD command executing copy in serial mode even if parallel execution is > enabled using WITH clause > --- > > Key: HIVE-21055 > URL: https://issues.apache.org/jira/browse/HIVE-21055 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch, > HIVE-21055.03.patch > > > For repl load command use can specify the execution mode as part of "with" > clause. But the config for executing task in parallel or serial is not read > from the command specific config. It is read from the hive server config. So > even if user specifies to run the tasks in parallel during repl load command, > the tasks are getting executed serially. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21144) ODBC with prepared statement fail inside a CTE
[
https://issues.apache.org/jira/browse/HIVE-21144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766055#comment-16766055
]
Guillaume commented on HIVE-21144:
--
I'm pretty sure that you are right. And there is no way to report a bug to
Simba or Hortonworks if I am not a paying customer, so I'm pretty much out of
luck. Thanks!
> ODBC with prepared statement fail inside a CTE
> --
>
> Key: HIVE-21144
> URL: https://issues.apache.org/jira/browse/HIVE-21144
> Project: Hive
> Issue Type: Bug
> Components: ODBC
>Affects Versions: 3.1.0
>Reporter: Guillaume
>Priority: Major
>
> I am trying to execute a very simple query, using python/pyodbc on Windows
> (with a working system-wide odbc DSN: HiveProd):
> {code:java}
> import pyodbc cnxn = pyodbc.connect('DSN=HiveProd', autocommit=True)
> cursor = cnxn.cursor()
> # works
> q="select ? as lic, ? as cpg"
> # fails
> q="with init as (select ? as lic, ? as cpg) select * from init"
> cursor.execute(q, '1', 'some string')
> for row in cursor:
> print(row.lic, row.cpg)
> {code}
> Basically, create an odbc connection, run a query with a prepared statement
> and print the result.
> A basic query works fine. If I put this query inside a CTE, I get:
> {{ cursor.execute("with init as (select ? as lic, ? as cpg) select * from
> init", '1', 'some string') pyodbc.ProgrammingError: ('42000', "[42000]
> [Hortonworks][Hardy] (80) Syntax or semantic analysis error thrown in server
> while executing query. Error message from server: Error while compiling
> statement: FAILED: ParseException line 1:21 can not recognize input near '?'
> 'as' 'lic' in select clause (80) (SQLPrepare)")}}
> This is not specific to python as I get the same issue with .Net.
> Trying the same with JDBC works fine.
> Testing on Hive 3.1.0 from Hdp 3.1.0
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[
https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766025#comment-16766025
]
Hive QA commented on HIVE-21244:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2295 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 6 unchanged - 0 fixed
= 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16033/dev-support/hive-personality.sh
|
| git revision | master / e540cf8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16033/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16033/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NPE in Hive Proto Logger
>
>
> Key: HIVE-21244
> URL: https://issues.apache.org/jira/browse/HIVE-21244
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-21244.1.patch, HIVE-21244.2.patch
>
>
> [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308]
> can cause NPE. There is no uncaught exception handler for this thread. This
> NPE can silently fail and drop the event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Patch Available (was: Open) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Description:
Ultimately, I'd like to get rid of
{{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
make it easier to get rid of later. It's currently buried in
{{PlanUtils.java}}.
A SerDe and a boolean flag gets passed into these methods. If the flag is set
to true, the specified SerDe is overwritten and assigned to
{{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird thing
to do, just pass in the required SerDe from the start instead of sending the
wrong SerDe and a flag to overwrite it.
was:
Ultimately, I'd like to get rid of
{{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
make it easier to get rid of later. It's currently buried in
{{PlanUtils.java}}.
A SerDe and a flag gets passed into utilities. If the class boolean is set,
the passed-in SerDe is overwritten. This is not documented anywhere and it's
weird to do it, just pass in the SerDe to use instead of the SerDe you don't
want to use and a flag to change it.
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a boolean flag gets passed into these methods. If the flag is
> set to true, the specified SerDe is overwritten and assigned to
> {{DelimitedJSONSerDe}}. This is not documented anywhere and it's a weird
> thing to do, just pass in the required SerDe from the start instead of
> sending the wrong SerDe and a flag to overwrite it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Status: Open (was: Patch Available) > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.1.1, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21240) JSON SerDe Deserialize Re-Write
[ https://issues.apache.org/jira/browse/HIVE-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-21240: --- Attachment: HIVE-21240.1.patch > JSON SerDe Deserialize Re-Write > --- > > Key: HIVE-21240 > URL: https://issues.apache.org/jira/browse/HIVE-21240 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0, 3.1.1 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21240.1.patch, HIVE-21240.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The JSON SerDe has a few issues, I will link them to this JIRA. > * Use Jackson Tree parser instead of manually parsing > * Added support for base-64 encoded data (the expected format when using JSON) > * Added support to skip blank lines (returns all columns as null values) > * Current JSON parser accepts, but does not apply, custom timestamp formats > in most cases > * Added some unit tests > * Added cache for column-name to column-index searches, currently O\(n\) for > each row processed, for each column in the row -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Attachment: HIVE-21246.1.patch
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Status: Patch Available (was: Open)
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21246) Un-bury DelimitedJSONSerDe from PlanUtils.java
[
https://issues.apache.org/jira/browse/HIVE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-21246:
---
Status: Open (was: Patch Available)
> Un-bury DelimitedJSONSerDe from PlanUtils.java
> --
>
> Key: HIVE-21246
> URL: https://issues.apache.org/jira/browse/HIVE-21246
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-21246.1.patch, HIVE-21246.1.patch
>
>
> Ultimately, I'd like to get rid of
> {{org.apache.hadoop.hive.serde2.DelimitedJSONSerDe}}, but for now, trying to
> make it easier to get rid of later. It's currently buried in
> {{PlanUtils.java}}.
> A SerDe and a flag gets passed into utilities. If the class boolean is set,
> the passed-in SerDe is overwritten. This is not documented anywhere and it's
> weird to do it, just pass in the SerDe to use instead of the SerDe you don't
> want to use and a flag to change it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21241) Migrate TimeStamp Parser From Joda Time
[
https://issues.apache.org/jira/browse/HIVE-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766010#comment-16766010
]
BELUGA BEHR commented on HIVE-21241:
[~ngangam] [~pvary] Can you please take a look at this? :)
> Migrate TimeStamp Parser From Joda Time
> ---
>
> Key: HIVE-21241
> URL: https://issues.apache.org/jira/browse/HIVE-21241
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21241.1.patch, HIVE-21241.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive uses Joda time for its TimeStampParser.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Migrate TimeStampParser to {{java.time}}
> I also added a couple new pre-canned timestamp parsers for convenience:
> * ISO 8601
> * RFC 1123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21055) Replication load command executing copy in serial mode even if parallel execution is enabled using with clause
[
https://issues.apache.org/jira/browse/HIVE-21055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765999#comment-16765999
]
Hive QA commented on HIVE-21055:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958340/HIVE-21055.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15786 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16032/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16032/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16032/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12958340 - PreCommit-HIVE-Build
> Replication load command executing copy in serial mode even if parallel
> execution is enabled using with clause
> --
>
> Key: HIVE-21055
> URL: https://issues.apache.org/jira/browse/HIVE-21055
> Project: Hive
> Issue Type: Bug
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21055.01.patch, HIVE-21055.02.patch,
> HIVE-21055.03.patch
>
>
> For repl load command use can specify the execution mode as part of "with"
> clause. But the config for executing task in parallel or serial is not read
> from the command specific config. It is read from the hive server config. So
> even if user specifies to run the tasks in parallel during repl load command,
> the tasks are getting executed serially.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-21250: --- Status: Patch Available (was: Reopened) > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765985#comment-16765985 ] anishek commented on HIVE-21250: +1 looks good to me, pending test > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
