[jira] [Commented] (HIVE-19130) NPE is thrown when REPL LOAD applied drop partition event.
[
https://issues.apache.org/jira/browse/HIVE-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436934#comment-16436934
]
Sankar Hariappan commented on HIVE-19130:
-
This patch is backported to apache/branch-3 as well.
> NPE is thrown when REPL LOAD applied drop partition event.
> --
>
> Key: HIVE-19130
> URL: https://issues.apache.org/jira/browse/HIVE-19130
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19130.01.patch
>
>
> During incremental replication, if we split the events batch as follows, then
> the REPL LOAD on second batch throws NPE.
> Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)
> Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) ->
> DROP_PARTITION (t1.p1)
> {code}
> 2018-04-05 16:20:36,531 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> metadata.Hive (Hive.java:getTable(1219)) - Table catalog_sales_new not found:
> new5_tpcds_real_bin_partitioned_orc_1000.catalog_sales_new table not found
> 2018-04-05 16:20:36,538 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> exec.DDLTask (DDLTask.java:failed(540)) -
> org.apache.hadoop.hive.ql.metadata.HiveException
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4016)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3983)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:341)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:162)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1765)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1506)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByExpr(Hive.java:2613)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4008)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19193) TestActivePassiveHA fails
[
https://issues.apache.org/jira/browse/HIVE-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19193:
-
Status: Patch Available (was: Open)
> TestActivePassiveHA fails
> -
>
> Key: HIVE-19193
> URL: https://issues.apache.org/jira/browse/HIVE-19193
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Ashutosh Chauhan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19193.1.patch
>
>
> This looks like a flaky test. If its not reproducible locally try improving
> on stability of this test, since it does fail randomly with Hive QA.
> {code:java}
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover(TestActivePassiveHA.java:356)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19193) TestActivePassiveHA fails
[
https://issues.apache.org/jira/browse/HIVE-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19193:
-
Attachment: HIVE-19193.1.patch
> TestActivePassiveHA fails
> -
>
> Key: HIVE-19193
> URL: https://issues.apache.org/jira/browse/HIVE-19193
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Ashutosh Chauhan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19193.1.patch
>
>
> This looks like a flaky test. If its not reproducible locally try improving
> on stability of this test, since it does fail randomly with Hive QA.
> {code:java}
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover(TestActivePassiveHA.java:356)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19193) TestActivePassiveHA fails
[
https://issues.apache.org/jira/browse/HIVE-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436977#comment-16436977
]
Prasanth Jayachandran commented on HIVE-19193:
--
The test will no wait for futures to get resolved before checking leader/not
leader status. Should make the tests more stable. [~ashutoshc] can you please
take a look?
> TestActivePassiveHA fails
> -
>
> Key: HIVE-19193
> URL: https://issues.apache.org/jira/browse/HIVE-19193
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Ashutosh Chauhan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19193.1.patch
>
>
> This looks like a flaky test. If its not reproducible locally try improving
> on stability of this test, since it does fail randomly with Hive QA.
> {code:java}
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover(TestActivePassiveHA.java:356)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init
[
https://issues.apache.org/jira/browse/HIVE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-19199:
--
Assignee: Gopal V
> ACID: DbTxnManager heartbeat-service needs static sync init
>
>
> Key: HIVE-19199
> URL: https://issues.apache.org/jira/browse/HIVE-19199
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
>
> {code}
> private synchronized void initHeartbeatExecutorService() {
> if (heartbeatExecutorService != null &&
> !heartbeatExecutorService.isShutdown()
> && !heartbeatExecutorService.isTerminated()) {
> return;
> }
> ...
> {code}
> heartbeatExecutorService is a static field, so this needs class sync, not
> object sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init
[
https://issues.apache.org/jira/browse/HIVE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-19199:
---
Affects Version/s: 3.1.0
> ACID: DbTxnManager heartbeat-service needs static sync init
>
>
> Key: HIVE-19199
> URL: https://issues.apache.org/jira/browse/HIVE-19199
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
>
> {code}
> private synchronized void initHeartbeatExecutorService() {
> if (heartbeatExecutorService != null &&
> !heartbeatExecutorService.isShutdown()
> && !heartbeatExecutorService.isTerminated()) {
> return;
> }
> ...
> {code}
> heartbeatExecutorService is a static field, so this needs class sync, not
> object sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init
[
https://issues.apache.org/jira/browse/HIVE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-19199:
---
Status: Patch Available (was: Open)
> ACID: DbTxnManager heartbeat-service needs static sync init
>
>
> Key: HIVE-19199
> URL: https://issues.apache.org/jira/browse/HIVE-19199
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19199.1.patch
>
>
> {code}
> private synchronized void initHeartbeatExecutorService() {
> if (heartbeatExecutorService != null &&
> !heartbeatExecutorService.isShutdown()
> && !heartbeatExecutorService.isTerminated()) {
> return;
> }
> ...
> {code}
> heartbeatExecutorService is a static field, so this needs class sync, not
> object sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init
[
https://issues.apache.org/jira/browse/HIVE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-19199:
---
Attachment: HIVE-19199.1.patch
> ACID: DbTxnManager heartbeat-service needs static sync init
>
>
> Key: HIVE-19199
> URL: https://issues.apache.org/jira/browse/HIVE-19199
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19199.1.patch
>
>
> {code}
> private synchronized void initHeartbeatExecutorService() {
> if (heartbeatExecutorService != null &&
> !heartbeatExecutorService.isShutdown()
> && !heartbeatExecutorService.isTerminated()) {
> return;
> }
> ...
> {code}
> heartbeatExecutorService is a static field, so this needs class sync, not
> object sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19199) ACID: DbTxnManager heartbeat-service needs static sync init
[
https://issues.apache.org/jira/browse/HIVE-19199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-19199:
---
Component/s: Transactions
> ACID: DbTxnManager heartbeat-service needs static sync init
>
>
> Key: HIVE-19199
> URL: https://issues.apache.org/jira/browse/HIVE-19199
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19199.1.patch
>
>
> {code}
> private synchronized void initHeartbeatExecutorService() {
> if (heartbeatExecutorService != null &&
> !heartbeatExecutorService.isShutdown()
> && !heartbeatExecutorService.isTerminated()) {
> return;
> }
> ...
> {code}
> heartbeatExecutorService is a static field, so this needs class sync, not
> object sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19187) Update Druid Storage Handler to Druid 0.12.0
[
https://issues.apache.org/jira/browse/HIVE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437011#comment-16437011
]
Hive QA commented on HIVE-19187:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
3s{color} | {color:red} root: The patch generated 3 new + 188 unchanged - 3
fixed = 191 total (was 191) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} druid-handler: The patch generated 3 new + 188
unchanged - 3 fixed = 191 total (was 191) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10179/dev-support/hive-personality.sh
|
| git revision | master / 6f9090c |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10179/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10179/yetus/diff-checkstyle-druid-handler.txt
|
| modules | C: . druid-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10179/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Update Druid Storage Handler to Druid 0.12.0
>
>
> Key: HIVE-19187
> URL: https://issues.apache.org/jira/browse/HIVE-19187
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19187.patch
>
>
> Current used Druid Version is 0.11.0
> This Patch updates the Druid version to the most recent version 0.12.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19201) Hive doesn't read Druid data correctly
[ https://issues.apache.org/jira/browse/HIVE-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tournadre updated HIVE-19201: - Attachment: DruidIntegration.tar > Hive doesn't read Druid data correctly > -- > > Key: HIVE-19201 > URL: https://issues.apache.org/jira/browse/HIVE-19201 > Project: Hive > Issue Type: Bug > Components: Druid integration, Hive >Affects Versions: 2.3.0 > Environment: Ubuntu 16.4 TLS Desktop > Druid 0.12.0 (StandAlone - Quickstart conf) > Hive 2.3.0 (StandAlone - Quickstart conf) > Also have Hadoop and Zookeeper running >Reporter: Tournadre >Priority: Blocker > Attachments: DruidIntegration.tar > > > I created an external table on hive pointing at my datasource : wikiticker on > druid (I already defined the broker adress). However few colums appear as > NULL without knowing why. > I have also many time *.lck file in the metadatadb on Hive and Druid to > delete (otherwise, queries don't work) > So the describe statement lists the column and their type correctly, however > I only get "NULL" when I query some column (string type). I checked in my > Druid, the data are still here. > > Help ! :( > PS : Sorry first issue and JIRA, didn't see my issue resolved yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19187) Update Druid Storage Handler to Druid 0.12.0
[
https://issues.apache.org/jira/browse/HIVE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437051#comment-16437051
]
Hive QA commented on HIVE-19187:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918653/HIVE-19187.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 13401 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2.q,l
[jira] [Updated] (HIVE-19158) Fix NPE in the HiveMetastore add partition tests
[ https://issues.apache.org/jira/browse/HIVE-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19158: - Attachment: HIVE-19158.1.patch > Fix NPE in the HiveMetastore add partition tests > > > Key: HIVE-19158 > URL: https://issues.apache.org/jira/browse/HIVE-19158 > Project: Hive > Issue Type: Bug > Components: Metastore, Test >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19158.1.patch > > > The TestAddPartitions and TestAddPartitionsFromPartSpec tests revealed that > NPE is thrown in some cases. These NPEs could be prevented with a simple null > check and a MetaException with a proper error message should be thrown > instead. > Example: NPE is thrown in the following test cases > * TestAddPartitions > ** testAddPartitionNullPartition > ** testAddPartitionNullValue > ** testAddPartitionsNullList > * TestAddPartitionsFromPartSpec > ** testAddPartitionSpecNullSpec > ** testAddPartitionSpecNullPartList > ** testAddPartitionSpecNoDB > ** testAddPartitionSpecNoTable > ** testAddPartitionSpecNoDBAndTableInPartition > ** testAddPartitionSpecNullPart > ** testAddPartitionSpecChangeRootPathToNull > ** testAddPartitionSpecWithSharedSDNullSd > ** testAddPartitionSpecWithSharedSDNoValue > ** testAddPartitionSpecNullValue -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18469) HS2UI: Introduce separate option to show query on web ui
[
https://issues.apache.org/jira/browse/HIVE-18469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437070#comment-16437070
]
Zoltan Haindrich commented on HIVE-18469:
-
great :)
+1
> HS2UI: Introduce separate option to show query on web ui
>
>
> Key: HIVE-18469
> URL: https://issues.apache.org/jira/browse/HIVE-18469
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18469.1.patch
>
>
> currently {{ConfVars.HIVE_LOG_EXPLAIN_OUTPUT}} enables 2 features:
> * log the query to the console (even thru beeline)
> * shows the query on the web ui
> I've enabled it...and ever since then my beeline is always flooded with an
> {{explain extended}} output...which is very verbose; even for simple queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19201) Hive doesn't read Druid data correctly
[ https://issues.apache.org/jira/browse/HIVE-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tournadre updated HIVE-19201: - Due Date: 17/Apr/18 (was: 9/Apr/18) > Hive doesn't read Druid data correctly > -- > > Key: HIVE-19201 > URL: https://issues.apache.org/jira/browse/HIVE-19201 > Project: Hive > Issue Type: Bug > Components: Druid integration, Hive >Affects Versions: 2.3.0 > Environment: Ubuntu 16.4 TLS Desktop > Druid 0.12.0 (StandAlone - Quickstart conf) > Hive 2.3.0 (StandAlone - Quickstart conf) > Also have Hadoop and Zookeeper running >Reporter: Tournadre >Priority: Blocker > Attachments: DruidIntegration.tar > > > I created an external table on hive pointing at my datasource : wikiticker on > druid (I already defined the broker adress). However few colums appear as > NULL without knowing why. > I have also many time *.lck file in the metadatadb on Hive and Druid to > delete (otherwise, queries don't work) > So the describe statement lists the column and their type correctly, however > I only get "NULL" when I query some column (string type). I checked in my > Druid, the data are still here. > > Help ! :( > PS : Sorry first issue and JIRA, didn't see my issue resolved yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437077#comment-16437077
]
Hive QA commented on HIVE-18902:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10181/dev-support/hive-personality.sh
|
| git revision | master / 6f9090c |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10181/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassi
[jira] [Commented] (HIVE-19133) HS2 WebUI phase-wise performance metrics not showing correctly
[ https://issues.apache.org/jira/browse/HIVE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437083#comment-16437083 ] Zoltan Haindrich commented on HIVE-19133: - +1 > HS2 WebUI phase-wise performance metrics not showing correctly > -- > > Key: HIVE-19133 > URL: https://issues.apache.org/jira/browse/HIVE-19133 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Web UI >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-19133.1.patch, HIVE-19133.2.patch, > HIVE-19133.3.patch, WebUI-compile time query metrics.png > > > The query specific WebUI metrics (go to drilldown -> performance logging) are > not showing up in the correct phase and are often mixed up. > Attaching screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()
[ https://issues.apache.org/jira/browse/HIVE-19202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei reassigned HIVE-19202: - > CBO failed due to NullPointerException in HiveAggregate.isBucketedInput() > - > > Key: HIVE-19202 > URL: https://issues.apache.org/jira/browse/HIVE-19202 > Project: Hive > Issue Type: Bug >Reporter: zhuwei >Assignee: zhuwei >Priority: Critical > > I ran a query with join and group by with below settings, COB failed due to > NullPointerException in HiveAggregate.isBucketedInput() > set hive.execution.engine=tez; > set hive.cbo.costmodel.extended=true; > > In class HiveRelMdDistribution, we implemented below functions: > public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery > mq) > public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq) > > But in HiveAggregate.isBucketedInput, the argument passed to distribution is > "this.getInput()" > , obviously it's not right here. The right argument needed is "this" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Assignee: Matt McCline Status: Patch Available (was: Open) > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Attachment: HIVE-19200.01.patch > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Fix Version/s: 3.0.0 > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437170#comment-16437170
]
Bohdan Chupika commented on HIVE-18902:
---
[~belugabehr] Could you review please?
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437171#comment-16437171
]
Hive QA commented on HIVE-18902:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918691/HIVE-18902.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 14143 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=71)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[default_constraint_invalid_default_value_type]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_list_bucketing]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_nonexistant_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_max]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_min]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_next_day_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_next_day_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_nonexistent_resource]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_by_wrong1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_wrong1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_wrong2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_sort_array_wrong3]
(batchId=96)
org.ap
[jira] [Updated] (HIVE-17477) PidFilePatternConverter not rolling log according to their pid
[
https://issues.apache.org/jira/browse/HIVE-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bohdan Chupika updated HIVE-17477:
--
Attachment: HIVE-17477.2.patch
> PidFilePatternConverter not rolling log according to their pid
> ---
>
> Key: HIVE-17477
> URL: https://issues.apache.org/jira/browse/HIVE-17477
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Vlad Gudikov
>Assignee: Bohdan Chupika
>Priority: Major
> Attachments: HIVE-17477.1.patch, HIVE-17477.2.patch
>
>
> We can use pid in th filePattern:
> appender.DRFA.filePattern =
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}.%pid
> But when its time roll the logs at the end of the RollingAppender just
> renames hive.log by applying the pattern described above and keeps logs that
> are not related to the process this log was named by. The issue is that all
> processes are writing to the same log and we cannot separate them by pid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17477) PidFilePatternConverter not rolling log according to their pid
[
https://issues.apache.org/jira/browse/HIVE-17477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437182#comment-16437182
]
Bohdan Chupika commented on HIVE-17477:
---
HIVE-17477.2.patch fixed checkstyle
> PidFilePatternConverter not rolling log according to their pid
> ---
>
> Key: HIVE-17477
> URL: https://issues.apache.org/jira/browse/HIVE-17477
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Vlad Gudikov
>Assignee: Bohdan Chupika
>Priority: Major
> Attachments: HIVE-17477.1.patch, HIVE-17477.2.patch
>
>
> We can use pid in th filePattern:
> appender.DRFA.filePattern =
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}.%pid
> But when its time roll the logs at the end of the RollingAppender just
> renames hive.log by applying the pattern described above and keeps logs that
> are not related to the process this log was named by. The issue is that all
> processes are writing to the same log and we cannot separate them by pid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed
[
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437190#comment-16437190
]
Hive QA commented on HIVE-16041:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hcatalog/core: The patch generated 1 new + 50
unchanged - 0 fixed = 51 total (was 50) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10183/dev-support/hive-personality.sh
|
| git revision | master / 6f9090c |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10183/yetus/diff-checkstyle-hcatalog_core.txt
|
| modules | C: hcatalog/core U: hcatalog/core |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10183/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HCatalog doesn't delete temp _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-16041.1.patch, HIVE-16041.2.patch,
> HIVE-16041.3.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table, a
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the
> job failed.
> {quote}
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:45
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:51
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x - yun hdfs 0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x - yun hdfs 0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r-- 3 yun hdfs 12 2017-02-27 00:57
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r-- 3 yun hdfs 12 2017-02-27 01:28
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19158) Fix NPE in the HiveMetastore add partition tests
[ https://issues.apache.org/jira/browse/HIVE-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437223#comment-16437223 ] Peter Vary commented on HIVE-19158: --- +1 pending tests > Fix NPE in the HiveMetastore add partition tests > > > Key: HIVE-19158 > URL: https://issues.apache.org/jira/browse/HIVE-19158 > Project: Hive > Issue Type: Bug > Components: Metastore, Test >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19158.1.patch > > > The TestAddPartitions and TestAddPartitionsFromPartSpec tests revealed that > NPE is thrown in some cases. These NPEs could be prevented with a simple null > check and a MetaException with a proper error message should be thrown > instead. > Example: NPE is thrown in the following test cases > * TestAddPartitions > ** testAddPartitionNullPartition > ** testAddPartitionNullValue > ** testAddPartitionsNullList > * TestAddPartitionsFromPartSpec > ** testAddPartitionSpecNullSpec > ** testAddPartitionSpecNullPartList > ** testAddPartitionSpecNoDB > ** testAddPartitionSpecNoTable > ** testAddPartitionSpecNoDBAndTableInPartition > ** testAddPartitionSpecNullPart > ** testAddPartitionSpecChangeRootPathToNull > ** testAddPartitionSpecWithSharedSDNullSd > ** testAddPartitionSpecWithSharedSDNoValue > ** testAddPartitionSpecNullValue -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Attachment: HIVE-19200.01.patch > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Attachment: (was: HIVE-19200.01.patch) > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed
[
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437248#comment-16437248
]
Hive QA commented on HIVE-16041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918704/HIVE-16041.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 13798 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=96)
[udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,load_part_nospec.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,msck_repair_1.q,orc_change_fileformat_acid.q,udf_nonexistent_resource.q,msck_repair_3.q,exim_19_external_over_existing.q,serde_r
[jira] [Updated] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()
[ https://issues.apache.org/jira/browse/HIVE-19202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19202: -- Attachment: HIVE-19202.1.patch > CBO failed due to NullPointerException in HiveAggregate.isBucketedInput() > - > > Key: HIVE-19202 > URL: https://issues.apache.org/jira/browse/HIVE-19202 > Project: Hive > Issue Type: Bug >Reporter: zhuwei >Assignee: zhuwei >Priority: Critical > Attachments: HIVE-19202.1.patch > > > I ran a query with join and group by with below settings, COB failed due to > NullPointerException in HiveAggregate.isBucketedInput() > set hive.execution.engine=tez; > set hive.cbo.costmodel.extended=true; > > In class HiveRelMdDistribution, we implemented below functions: > public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery > mq) > public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq) > > But in HiveAggregate.isBucketedInput, the argument passed to distribution is > "this.getInput()" > , obviously it's not right here. The right argument needed is "this" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19202) CBO failed due to NullPointerException in HiveAggregate.isBucketedInput()
[ https://issues.apache.org/jira/browse/HIVE-19202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19202: -- Fix Version/s: 2.1.1 Affects Version/s: 2.1.1 Status: Patch Available (was: Open) > CBO failed due to NullPointerException in HiveAggregate.isBucketedInput() > - > > Key: HIVE-19202 > URL: https://issues.apache.org/jira/browse/HIVE-19202 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Critical > Fix For: 2.1.1 > > Attachments: HIVE-19202.1.patch > > > I ran a query with join and group by with below settings, COB failed due to > NullPointerException in HiveAggregate.isBucketedInput() > set hive.execution.engine=tez; > set hive.cbo.costmodel.extended=true; > > In class HiveRelMdDistribution, we implemented below functions: > public RelDistribution distribution(HiveAggregate aggregate, RelMetadataQuery > mq) > public RelDistribution distribution(HiveJoin join, RelMetadataQuery mq) > > But in HiveAggregate.isBucketedInput, the argument passed to distribution is > "this.getInput()" > , obviously it's not right here. The right argument needed is "this" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18946:
Attachment: (was: HIVE-18946.03.patch)
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18946:
Attachment: HIVE-18946.03.patch
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18946:
Attachment: HIVE-18946.03.patch
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch,
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437295#comment-16437295
]
Laszlo Bodor commented on HIVE-18946:
-
[~ashutoshc], [~kgyrtkirk] : i've uploaded 03.patch which handles flipped
values, and covers some cases with unit tests.
The Decimal.compareTo issue will be handled HIVE-19131
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch,
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17838) Make org.apache.hive.spark.client.rpc logging HoS specific and other logging cleanup
[
https://issues.apache.org/jira/browse/HIVE-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17838:
Attachment: HIVE-17838.2.patch
> Make org.apache.hive.spark.client.rpc logging HoS specific and other logging
> cleanup
>
>
> Key: HIVE-17838
> URL: https://issues.apache.org/jira/browse/HIVE-17838
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17838.1.patch, HIVE-17838.2.patch
>
>
> A lot of the logging in {{org.apache.hive.spark.client.rpc}} emits
> information that a generic RPC library would, but this package is only used
> by HoS. We should make the logging more HoS specific so users don't get
> confused as to what this RPC library is doing. Ideally, they aren't even
> aware that a RPC library is in place, it should be more transparent to the
> user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437300#comment-16437300
]
Hive QA commented on HIVE-12369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
23s{color} | {color:red} root: The patch generated 320 new + 1088 unchanged -
51 fixed = 1408 total (was 1139) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 288 new + 356 unchanged - 42
fixed = 644 total (was 398) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} vector-code-gen: The patch generated 32 new + 307
unchanged - 9 fixed = 339 total (was 316) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10184/dev-support/hive-personality.sh
|
| git revision | master / 6f9090c |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus/diff-checkstyle-vector-code-gen.txt
|
| modules | C: common . ql vector-code-gen U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10184/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Native Vector GroupBy
> -
>
> Key: HIVE-12369
> URL: https://issues.apache.org/jira/browse/HIVE-12369
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12369.01.patch, HIVE-12369.02.patch,
> HIVE-12369.05.patch, HIVE-12369.06.patch, HIVE-12369.091.patch,
> HIVE-12369.094.patch
>
>
> Implement Native Vector GroupBy using fast hash table technology developed
> for Native Vector MapJoin, etc.
> Patc
[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18946:
Attachment: (was: HIVE-18946.03.patch)
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch,
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-18946:
Attachment: HIVE-18946.03.patch
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch,
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437320#comment-16437320
]
Zoltan Haindrich commented on HIVE-18946:
-
+1
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch,
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18902) Lower Logging Level for Cleaning Up "local RawStore"
[
https://issues.apache.org/jira/browse/HIVE-18902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437328#comment-16437328
]
BELUGA BEHR commented on HIVE-18902:
[~bchupika] Thanks. :)
I think what we should do here is perhaps is simply to modify the {{logInfo}}
method to be called {{logAndAudit}} and then simply change the logging level
here from 'info' to 'debug'. The message will be logged at the 'info' level in
the {{logAuditEvent}} method, so it seems superfluous to log it twice, in two
locations, at the 'info' level.
{code:java}
private static void logAndAudit(String m) {
LOG.debug("{}: {}", threadLocalId.get(), m);
logAuditEvent(m);
}{code}
> Lower Logging Level for Cleaning Up "local RawStore"
>
>
> Key: HIVE-18902
> URL: https://issues.apache.org/jira/browse/HIVE-18902
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Bohdan Chupika
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18902.01.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L7756-L7768]
>
> {code:java}
> private static void cleanupRawStore() {
> try {
> RawStore rs = HMSHandler.getRawStore();
> if (rs != null) {
> HMSHandler.logInfo("Cleaning up thread local RawStore...");
> rs.shutdown();
> }
> } finally {
> HMSHandler handler = HMSHandler.threadLocalHMSHandler.get();
> if (handler != null) {
> handler.notifyMetaListenersOnShutDown();
> }
> HMSHandler.threadLocalHMSHandler.remove();
> HMSHandler.threadLocalConf.remove();
> HMSHandler.threadLocalModifiedConfig.remove();
> HMSHandler.removeRawStore();
> HMSHandler.logInfo("Done cleaning up thread local RawStore");
> }
> }
> {code}
> {code}
> 2018-03-03 17:21:49,832 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19:
> Cleaning up thread local RawStore...
> 2018-03-03 17:21:49,834 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-4-thread-21]: 19: Done
> cleaning up thread local RawStore
> {code}
> Not very helpful logging. Please change logging levels to _debug_ or even
> _trace_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18946) Fix columnstats merge NPE
[
https://issues.apache.org/jira/browse/HIVE-18946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18946:
Target Version/s: 3.0.0
> Fix columnstats merge NPE
> -
>
> Key: HIVE-18946
> URL: https://issues.apache.org/jira/browse/HIVE-18946
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18946.01.patch, HIVE-18946.02.patch,
> HIVE-18946.03.patch
>
>
> after analyzing an empty table may lead to an NPE when inserting into it...
> {code}
> 2018-03-13T06:54:22,503 ERROR [df3fb505-e0bc-4595-a874-b735dab8dff6 main]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318)
> at
> org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:778)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6934)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy55.set_aggr_stats_for(Unknown Source)
> [...]
> {code}
> reproduce
> {code}
> set hive.stats.autogather=true;
> set hive.explain.user=true;
> drop table if exists testdeci2;
> create table testdeci2(
> id int,
> amount decimal(10,3),
> sales_tax decimal(10,3),
> item string)
> stored as orc location '/tmp/testdeci2'
> TBLPROPERTIES ("transactional"="false")
> ;
> analyze table testdeci2 compute statistics for columns;
> insert into table testdeci2
> values(1,12.123,12345.123,'desk1'),(2,123.123,1234.123,'desk2');
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19131) DecimalColumnStatsMergerTest comparison review
[
https://issues.apache.org/jira/browse/HIVE-19131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-19131:
Target Version/s: 3.0.0
> DecimalColumnStatsMergerTest comparison review
> --
>
> Key: HIVE-19131
> URL: https://issues.apache.org/jira/browse/HIVE-19131
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> DecimalColumnStatsMergerTest has a strange comparison logic, which needs to
> be reviewed.
> Regarding low and high values, it uses compareTo with the same direction,
> which seems to be incorrect: old.compareTo(new) > 0 -> pick old value in both
> cases
> {code:java}
> Decimal lowValue = aggregateData.getLowValue() != null &&
> (aggregateData.getLowValue().compareTo(newData.getLowValue()) > 0) ?
> aggregateData .getLowValue() : newData.getLowValue();
> aggregateData.setLowValue(lowValue);
> Decimal highValue = aggregateData.getHighValue() != null &&
> (aggregateData.getHighValue().compareTo(newData.getHighValue()) > 0) ?
> aggregateData .getHighValue() : newData.getHighValue();
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12369) Native Vector GroupBy
[
https://issues.apache.org/jira/browse/HIVE-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437333#comment-16437333
]
Hive QA commented on HIVE-12369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918705/HIVE-12369.094.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 115 failed/errored test(s), 13799 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong
[jira] [Commented] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437374#comment-16437374 ] Peter Vary commented on HIVE-17318: --- This feature is not working, since MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" not "HikariCP" [~asinkovits]: If you have time, could you please check this out? > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17318.01.patch, HIVE-17318.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19194) TestDruidStorageHandler fails
[
https://issues.apache.org/jira/browse/HIVE-19194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437377#comment-16437377
]
slim bouguerra commented on HIVE-19194:
---
[~ashutoshc] do you have the rest of the log please?
> TestDruidStorageHandler fails
> -
>
> Key: HIVE-19194
> URL: https://issues.apache.org/jira/browse/HIVE-19194
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: slim bouguerra
>Priority: Major
>
> This tests fails randomly. If its not reproducible locally consider improving
> its stability since it does fail once in a while on Hive QA.
> {code}
> java.lang.AssertionError: expected:<0> but was:<1> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:743) at
> org.junit.Assert.assertEquals(Assert.java:118) at
> org.junit.Assert.assertEquals(Assert.java:555) at
> org.junit.Assert.assertEquals(Assert.java:542) at
> org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable(TestDruidStorageHandler.java:414)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12192) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437394#comment-16437394
]
Hive QA commented on HIVE-12192:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} druid-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hbase-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
5s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} druid-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} hbase-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} druid-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} hbase-handler in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 4 new + 35 unchanged
- 16 fixed = 39 total (was 51) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 6 new + 17 unchanged - 8
fixed = 23 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} serde: The patch generated 1 new + 2611 unchanged - 19
fixed = 2612 total (was 2630) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch druid-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hbase-handler: The patch generated 0 new + 18
unchanged - 1 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/hcatalog-pig-adapter: The patch generated 1
new + 85 unchanged - 1 fixed = 86 total (was 86) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hive-jmh passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
25s{color} | {color:red} ql: The patch generated 358 new + 5010 unchanged - 61
fixed = 5368 total (was 5071) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch vector-code-gen passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{colo
[jira] [Comment Edited] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437374#comment-16437374 ] Peter Vary edited comment on HIVE-17318 at 4/13/18 2:59 PM: This feature is not working, since MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" not "Hikari" [~asinkovits]: If you have time, could you please check this out? was (Author: pvary): This feature is not working, since MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" not "HikariCP" [~asinkovits]: If you have time, could you please check this out? > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17318.01.patch, HIVE-17318.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12192) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437463#comment-16437463
]
Hive QA commented on HIVE-12192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918739/HIVE-12192.03.patch
{color:green}SUCCESS:{color} +1 due to 47 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 403 failed/errored test(s), 13794 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong
[jira] [Commented] (HIVE-6980) Drop table by using direct sql
[
https://issues.apache.org/jira/browse/HIVE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437470#comment-16437470
]
Peter Vary commented on HIVE-6980:
--
[~selinazh]: do you plan to work on this? We have customers having problems
with long dropping times. If you do not have time, I would be happy to work on
this. If you have some pointers how were you able to convince DataNucleus to
leave the drops for the DB FK constraints it would be very helpful. I was
trying to do this by setting {{datanucleus.deletionPolicy}} to {{DataNucleus}}
but the log still shows that separate DN query is issued to drop the child data
even though the FK is present in the database, and working.
If we are not able to move forward with the cascade solution, then I could
provide a directsql solution instead, which is more straightforward
Thanks,
Peter
> Drop table by using direct sql
> --
>
> Key: HIVE-6980
> URL: https://issues.apache.org/jira/browse/HIVE-6980
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 0.12.0
>Reporter: Selina Zhang
>Assignee: Selina Zhang
>Priority: Major
>
> Dropping table which has lots of partitions is slow. Even after applying the
> patch of HIVE-6265, the drop table still takes hours (100K+ partitions).
> The fixes come with two parts:
> 1. use directSQL to query the partitions protect mode;
> the current implementation needs to transfer the Partition object to client
> and check the protect mode for each partition. I'd like to move this part of
> logic to metastore. The check will be done by direct sql (if direct sql is
> disabled, execute the same logic in the ObjectStore);
> 2. use directSQL to drop partitions for table;
> there maybe two solutions here:
> 1. add "DELETE CASCADE" in the schema. In this way we only need to delete
> entries from partitions table use direct sql. May need to change
> datanucleus.deletionPolicy = DataNucleus.
> 2. clean up the dependent tables by issue DELETE statement. This also needs
> to turn on datanucleus.query.sql.allowAll
> Both of above solutions should be able to fix the problem. The DELETE CASCADE
> has to change schemas and prepare upgrade scripts. The second solutions added
> maintenance cost if new tables added in the future releases.
> Please advice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Description: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. was:Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The longer-term solution can be done later in steps: > 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert > columns to different data types. This is needed when LLAP is doing its > encoding / caching and feeds VRBs to VectorMapOperator instead of rows. > Similar to what MapOperator does today, VectorMapOperator would need to be > enhanced to convert partition VRBs into the table schema VRBs that the vector > operator tree expect. > 2) Today, vectorization logic is strictly positional based. It insists that > the partition columns have the same names as the table schema. The > MapOperator (and ORC) does more general conversion that uses column names > instead of column position. We'd need to enhance all 3 classes to handle > column name based conversion. The 3 classes are: the new VRB-to-VRB > conversion class, VectorDeserializeRow, and VectorAssignRow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Description: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The main changes are: 1) In the Vectorizer, detect if data type conversion is needed between the partition and the desired table schema. If so and LLAP I/O is enabled that does encoded catching, then do not vectorize. Why? When LLAP I/O is in encoded catching mode, it delivers VectorizedRowBatch (VRB) to the VectorMapOperator instead of (object) rows. We currently do not have logic for converting VRBs. So, we either get Wrong Results or more likely ClassCastException on the expected vs actual ColumnVector columns. 2) Cleaned up error message logic.that was suppressing the new message from the EXPLAIN VECTORIZATION display. --- The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. was: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The main changes are: > 1) In the Vectorizer, detect if data type conversion is needed between the > partition and the desired table schema. If so and LLAP I/O is enabled that > does encoded catching, then do not vectorize. Why? When LLAP I/O is in > encoded catching mode, it delivers VectorizedRowBatch (VRB) to the > VectorMapOperator instead of (object) rows. We currently do not have logic > for converting VRBs. So, we either get Wrong Results or more likely > ClassCastException on the expected vs actual ColumnVector columns. > 2) Cleaned up error message logic.that was suppressing the new message from > the EXPLAIN VECTORIZATION display. > > --- > The longer-term solution can be done later in steps: > 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert > columns to different data types. This is needed when LLAP is doing its > encoding / caching and feeds VRBs to VectorMapOperator instead of rows. > Similar to what MapOperator does today, VectorMapOperator would need to be > enhanced
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Description: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The main changes are: 1) In the Vectorizer, detect if data type conversion is needed between the partition and the desired table schema. If so and LLAP I/O is enabled that does encoded catching, then do not vectorize. Why? When LLAP I/O is in encoded catching mode, it delivers VectorizedRowBatch (VRB) to the VectorMapOperator instead of (object) rows. We currently do not have logic for converting VRBs. So, we either get Wrong Results or more likely ClassCastException on the expected vs actual ColumnVector columns. 2) Cleaned up error message logic.that was suppressing the new message from the EXPLAIN VECTORIZATION display. 3) NOTE: Some of the SELECT statements int the schema_evol_test*.q are commented out because I bumped into a another bug. I'll file that one soon and add comments to the Q files. --- The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. was: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The main changes are: 1) In the Vectorizer, detect if data type conversion is needed between the partition and the desired table schema. If so and LLAP I/O is enabled that does encoded catching, then do not vectorize. Why? When LLAP I/O is in encoded catching mode, it delivers VectorizedRowBatch (VRB) to the VectorMapOperator instead of (object) rows. We currently do not have logic for converting VRBs. So, we either get Wrong Results or more likely ClassCastException on the expected vs actual ColumnVector columns. 2) Cleaned up error message logic.that was suppressing the new message from the EXPLAIN VECTORIZATION display. --- The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The main changes are: > 1) In the Vectorizer, detect if data type conversion is needed between the > partition and the desired table schema. If so and LLAP I/O is enabled that > does encoded catching, the
[jira] [Updated] (HIVE-19200) Vectorization: Disable vectorization for LLAP I/O when a non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type conversion is needed
[ https://issues.apache.org/jira/browse/HIVE-19200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19200: Description: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The main changes are: 1) In the Vectorizer, detect if data type conversion is needed between the partition and the desired table schema. If so and LLAP I/O is enabled that does encoded catching, then do not vectorize. Why? When LLAP I/O is in encoded catching mode, it delivers VectorizedRowBatch (VRB) to the VectorMapOperator instead of (object) rows. We currently do not have logic for converting VRBs. So, we either get Wrong Results or more likely ClassCastException on the expected vs actual ColumnVector columns. 2) Cleaned up error message logic.that was suppressing the new message from the EXPLAIN VECTORIZATION display. 3) NOTE: Some of the SELECT statements in the schema_evol_test*.q are commented out because I bumped into a another bug. I'll file that one soon and add comments to the Q files. --- The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. was: Disable vectorization for issue in HIVE-18763 until we can do the harder VRB conversion code. The main changes are: 1) In the Vectorizer, detect if data type conversion is needed between the partition and the desired table schema. If so and LLAP I/O is enabled that does encoded catching, then do not vectorize. Why? When LLAP I/O is in encoded catching mode, it delivers VectorizedRowBatch (VRB) to the VectorMapOperator instead of (object) rows. We currently do not have logic for converting VRBs. So, we either get Wrong Results or more likely ClassCastException on the expected vs actual ColumnVector columns. 2) Cleaned up error message logic.that was suppressing the new message from the EXPLAIN VECTORIZATION display. 3) NOTE: Some of the SELECT statements int the schema_evol_test*.q are commented out because I bumped into a another bug. I'll file that one soon and add comments to the Q files. --- The longer-term solution can be done later in steps: 1) Write a new code that can take a VectorizedRowBatch (VRB) and convert columns to different data types. This is needed when LLAP is doing its encoding / caching and feeds VRBs to VectorMapOperator instead of rows. Similar to what MapOperator does today, VectorMapOperator would need to be enhanced to convert partition VRBs into the table schema VRBs that the vector operator tree expect. 2) Today, vectorization logic is strictly positional based. It insists that the partition columns have the same names as the table schema. The MapOperator (and ORC) does more general conversion that uses column names instead of column position. We'd need to enhance all 3 classes to handle column name based conversion. The 3 classes are: the new VRB-to-VRB conversion class, VectorDeserializeRow, and VectorAssignRow. > Vectorization: Disable vectorization for LLAP I/O when a > non-VECTORIZED_INPUT_FILE_FORMAT mode is needed (i.e. rows) and data type > conversion is needed > --- > > Key: HIVE-19200 > URL: https://issues.apache.org/jira/browse/HIVE-19200 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19200.01.patch > > > Disable vectorization for issue in HIVE-18763 until we can do the harder VRB > conversion code. > The main changes are: > 1)
[jira] [Updated] (HIVE-19173) Add Storage Handler runtime information as part of DESCRIBE EXTENDED
[ https://issues.apache.org/jira/browse/HIVE-19173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19173: Attachment: HIVE-19173.01.patch > Add Storage Handler runtime information as part of DESCRIBE EXTENDED > > > Key: HIVE-19173 > URL: https://issues.apache.org/jira/browse/HIVE-19173 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19173.01.patch > > > Follow up for https://issues.apache.org/jira/browse/HIVE-18976 > Kafka Indexing Service in Druid has a runtime state associated with it. > Druid publishes this runtime state as KafkaSupervisorReport which has latest > offsets as reported by Kafka, the consumer lag per partition, as well as the > aggregate lag of all partitions. > This information is quite useful to know whether a kafka-indexing-service > backed table has latest info or not. > This task is to add a this information as part of the output of DESCRIBE > EXTENDED statement -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19173) Add Storage Handler runtime information as part of DESCRIBE EXTENDED
[ https://issues.apache.org/jira/browse/HIVE-19173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19173: Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Add Storage Handler runtime information as part of DESCRIBE EXTENDED > > > Key: HIVE-19173 > URL: https://issues.apache.org/jira/browse/HIVE-19173 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19173.01.patch > > > Follow up for https://issues.apache.org/jira/browse/HIVE-18976 > Kafka Indexing Service in Druid has a runtime state associated with it. > Druid publishes this runtime state as KafkaSupervisorReport which has latest > offsets as reported by Kafka, the consumer lag per partition, as well as the > aggregate lag of all partitions. > This information is quite useful to know whether a kafka-indexing-service > backed table has latest info or not. > This task is to add a this information as part of the output of DESCRIBE > EXTENDED statement -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19173) Add Storage Handler runtime information as part of DESCRIBE EXTENDED
[ https://issues.apache.org/jira/browse/HIVE-19173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437556#comment-16437556 ] Nishant Bangarwa commented on HIVE-19173: - [~ashutoshc] please review. > Add Storage Handler runtime information as part of DESCRIBE EXTENDED > > > Key: HIVE-19173 > URL: https://issues.apache.org/jira/browse/HIVE-19173 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19173.01.patch > > > Follow up for https://issues.apache.org/jira/browse/HIVE-18976 > Kafka Indexing Service in Druid has a runtime state associated with it. > Druid publishes this runtime state as KafkaSupervisorReport which has latest > offsets as reported by Kafka, the consumer lag per partition, as well as the > aggregate lag of all partitions. > This information is quite useful to know whether a kafka-indexing-service > backed table has latest info or not. > This task is to add a this information as part of the output of DESCRIBE > EXTENDED statement -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19187) Update Druid Storage Handler to Druid 0.12.0
[ https://issues.apache.org/jira/browse/HIVE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19187: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! [~vgarg] It will be good to have this in branch-3 as well. > Update Druid Storage Handler to Druid 0.12.0 > > > Key: HIVE-19187 > URL: https://issues.apache.org/jira/browse/HIVE-19187 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19187.patch > > > Current used Druid Version is 0.11.0 > This Patch updates the Druid version to the most recent version 0.12.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437575#comment-16437575
]
Hive QA commented on HIVE-18862:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m
43s{color} | {color:red} root: The patch generated 3 new + 2126 unchanged - 2
fixed = 2129 total (was 2128) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} itests/util: The patch generated 3 new + 168 unchanged
- 2 fixed = 171 total (was 170) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10186/dev-support/hive-personality.sh
|
| git revision | master / 6f9090c |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10186/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10186/yetus/diff-checkstyle-itests_util.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10186/yetus/whitespace-eol.txt
|
| modules | C: . accumulo-handler contrib hbase-handler itests/util ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10186/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
> Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch,
> HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch,
> HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch
>
>
> # Parse .q files for source table usage
> # Add needed dataset annotations
> # Remove create table statements from "q_test_init.sql" like files
> # Hand
[jira] [Commented] (HIVE-19172) NPE due to null EnvironmentContext in DDLTask
[
https://issues.apache.org/jira/browse/HIVE-19172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437574#comment-16437574
]
Vineet Garg commented on HIVE-19172:
Pushed to branch-3
> NPE due to null EnvironmentContext in DDLTask
> -
>
> Key: HIVE-19172
> URL: https://issues.apache.org/jira/browse/HIVE-19172
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19172.patch
>
>
> Stack Trace -
> {code}
> 2018-04-11T02:52:51,386 ERROR [5f2e24bf-ac93-4977-84fe-aa2c5f674ea4 main]
> exec.DDLTask: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3539)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:392)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19187) Update Druid Storage Handler to Druid 0.12.0
[ https://issues.apache.org/jira/browse/HIVE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437581#comment-16437581 ] Vineet Garg commented on HIVE-19187: Pushed to branch-3. > Update Druid Storage Handler to Druid 0.12.0 > > > Key: HIVE-19187 > URL: https://issues.apache.org/jira/browse/HIVE-19187 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19187.patch > > > Current used Druid Version is 0.11.0 > This Patch updates the Druid version to the most recent version 0.12.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19187) Update Druid Storage Handler to Druid 0.12.0
[ https://issues.apache.org/jira/browse/HIVE-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19187: --- Fix Version/s: (was: 3.1.0) 3.0.0 > Update Druid Storage Handler to Druid 0.12.0 > > > Key: HIVE-19187 > URL: https://issues.apache.org/jira/browse/HIVE-19187 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19187.patch > > > Current used Druid Version is 0.11.0 > This Patch updates the Druid version to the most recent version 0.12.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19172) NPE due to null EnvironmentContext in DDLTask
[
https://issues.apache.org/jira/browse/HIVE-19172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19172:
---
Fix Version/s: (was: 3.1.0)
3.0.0
> NPE due to null EnvironmentContext in DDLTask
> -
>
> Key: HIVE-19172
> URL: https://issues.apache.org/jira/browse/HIVE-19172
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19172.patch
>
>
> Stack Trace -
> {code}
> 2018-04-11T02:52:51,386 ERROR [5f2e24bf-ac93-4977-84fe-aa2c5f674ea4 main]
> exec.DDLTask: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3539)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:392)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437599#comment-16437599
]
Hive QA commented on HIVE-18862:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918759/HIVE-18862.08.patch
{color:green}SUCCESS:{color} +1 due to 2095 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 147 failed/errored test(s), 13798 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wro
[jira] [Updated] (HIVE-19170) Fix TestMiniDruidKafkaCliDriver
[
https://issues.apache.org/jira/browse/HIVE-19170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-19170:
Attachment: HIVE-19170.addendum.patch
> Fix TestMiniDruidKafkaCliDriver
> ---
>
> Key: HIVE-19170
> URL: https://issues.apache.org/jira/browse/HIVE-19170
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19170.01.patch, HIVE-19170.addendum.patch,
> HIVE-19170.patch
>
>
> added in HIVE-18976
> the property key {{druid.kafka.query.files}} doesn't exists in
> testconfiguration.properties.
> because of this TestMiniDruidKafkaCliDriver tries to run *all* qtests...which
> time out...and produce
> {code}
> TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
> out) (batchId=252)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18915) Better client logging when a HoS session can't be opened
[
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18915:
Affects Version/s: 3.0.0
> Better client logging when a HoS session can't be opened
>
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch, HIVE-18915.2.patch,
> HIVE-18915.3.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client
> for Spark session [id]}} when a HoS session can't be opened, would be better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18915) Better client logging when a HoS session can't be opened
[
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18915:
Target Version/s: (was: 3.0.0)
> Better client logging when a HoS session can't be opened
>
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch, HIVE-18915.2.patch,
> HIVE-18915.3.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client
> for Spark session [id]}} when a HoS session can't be opened, would be better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19009) Retain and use runtime statistics during hs2 lifetime
[
https://issues.apache.org/jira/browse/HIVE-19009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437651#comment-16437651
]
Hive QA commented on HIVE-19009:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 9 new + 587 unchanged - 1
fixed = 596 total (was 588) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10187/dev-support/hive-personality.sh
|
| git revision | master / 2a3a7d3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10187/yetus/diff-checkstyle-ql.txt
|
| modules | C: common itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10187/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Retain and use runtime statistics during hs2 lifetime
> -
>
> Key: HIVE-19009
> URL: https://issues.apache.org/jira/browse/HIVE-19009
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19009.01.patch, HIVE-19009.02.patch,
> HIVE-19009.03.patch, HIVE-19009.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-19204:
---
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-19077) Handle duplicate ptests requests standing in queue at the same time
[
https://issues.apache.org/jira/browse/HIVE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal reopened HIVE-19077:
---
[~szita] this is wrecking havoc on the ptests. Can you please undo this patch
and rework on it to remove the future instance so that the person waiting gets
a run and not kept waiting endlessly?
There are cases like mine where the patch is so large that I need to keep
rebasing and posting due to continuous changes.
> Handle duplicate ptests requests standing in queue at the same time
> ---
>
> Key: HIVE-19077
> URL: https://issues.apache.org/jira/browse/HIVE-19077
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19077.0.patch, HIVE-19077.1.patch,
> HIVE-19077.sslFix.patch
>
>
> I've been keeping on eye on our {{PreCommit-HIVE-Build}} job, and what I
> noticed that sometimes huge queues can build up, that contain jira's more
> than once. (Yesterday I've seen a queue of 40, having 31 distinct jiras..)
> Simple scenario is that I upload a patch, it gets queued for ptest (already
> long queue), and 3 hours later I will update it, re-upload and re-queue. Now
> the current ptest infra seems to be smart enough to always deal with the
> latest patch, so what will happen is that the same patch will be tested 2
> times (with ~3 hours) diff, most probably with same result.
> I propose we do some deduplication - if ptest starts running the request for
> Jira X, then it can take a look on the current queue, and see if X is there
> again. If so, it can skip for now, it will be picked up later anyway.
> In practice this means that if you reconsider your patch and update it, your
> original place in the queue will be gone (like as a penalty for changing it),
> but overall it saves resources for the whole community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19205: > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no compression, no > dictionary). Compactor will recreate the orc files with compression and > dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog > package for backward compatibility, new streaming API will be in its own hive > module) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19077) Handle duplicate ptests requests standing in queue at the same time
[
https://issues.apache.org/jira/browse/HIVE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Jaiswal updated HIVE-19077:
--
Priority: Blocker (was: Major)
> Handle duplicate ptests requests standing in queue at the same time
> ---
>
> Key: HIVE-19077
> URL: https://issues.apache.org/jira/browse/HIVE-19077
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19077.0.patch, HIVE-19077.1.patch,
> HIVE-19077.sslFix.patch
>
>
> I've been keeping on eye on our {{PreCommit-HIVE-Build}} job, and what I
> noticed that sometimes huge queues can build up, that contain jira's more
> than once. (Yesterday I've seen a queue of 40, having 31 distinct jiras..)
> Simple scenario is that I upload a patch, it gets queued for ptest (already
> long queue), and 3 hours later I will update it, re-upload and re-queue. Now
> the current ptest infra seems to be smart enough to always deal with the
> latest patch, so what will happen is that the same patch will be tested 2
> times (with ~3 hours) diff, most probably with same result.
> I propose we do some deduplication - if ptest starts running the request for
> Jira X, then it can take a look on the current queue, and see if X is there
> again. If so, it can skip for now, it will be picked up later anyway.
> In practice this means that if you reconsider your patch and update it, your
> original place in the queue will be gone (like as a penalty for changing it),
> but overall it saves resources for the whole community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19164) TestMetastoreVersion failures
[
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-19164:
--
Attachment: HIVE-19164.patch
> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws
> an exception {{No matching version found}}.
> To fetch schema version this method executes {code:sql} SELECT FROM
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever
> reason execution returns empty result set resulting in the exception. Both
> test failures are due to this exception. I tried debugging the query
> execution but didn't really go nowhere with it. I suspect this could be due
> to recent metastore changes. I tried reproducing this outside test but with
> no success.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19164) TestMetastoreVersion failures
[
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-19164:
--
Status: Patch Available (was: Open)
The attached patch fixes this, but is a big change to Driver's compile loop.
The patch changes Driver to pass its config file to Hive (the class) when it
calls Hive.get() to make sure Hive has the current config file. This is what
was causing the problem, as Hive had an old version of the config file from a
previous test and Driver was calling Hive.get() rather than Hive.get(conf).
This seems like what we want, in order to make sure Hive has the right config
file. However, I am not sure since Driver has access to its own config file,
the current session config file, and the current query config file. I passed
the Driver one, but I'm not sure that's the right choice. Also, this code has
not changed recently and may have other significant impacts since it's in
Driver's main compile loop.
cc [~ashutoshc] and [~kgyrtkirk] who I think know this code much better than I.
> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws
> an exception {{No matching version found}}.
> To fetch schema version this method executes {code:sql} SELECT FROM
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever
> reason execution returns empty result set resulting in the exception. Both
> test failures are due to this exception. I tried debugging the query
> execution but didn't really go nowhere with it. I suspect this could be due
> to recent metastore changes. I tried reproducing this outside test but with
> no success.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19164) TestMetastoreVersion failures
[
https://issues.apache.org/jira/browse/HIVE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437719#comment-16437719
]
Vineet Garg commented on HIVE-19164:
I think there were some discussions regarding this sometime ago and driver not
passing its own config to Hive was a deliberate decision. I might be
misremembering though. cc [~pvary] [~vihangk1] [~thejas] to see if they know
anything about this.
> TestMetastoreVersion failures
> -
>
> Key: HIVE-19164
> URL: https://issues.apache.org/jira/browse/HIVE-19164
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19164.patch
>
>
> Following tests are failing consistently and are reproducible on master:
> * testVersionMatching
> * testMetastoreVersion
> I tried debugging it and found that ObjectStore.getMSSchemaVersion() throws
> an exception {{No matching version found}}.
> To fetch schema version this method executes {code:sql} SELECT FROM
> org.apache.hadoop.hive.metastore.model.MVersionTable {code} but for whatever
> reason execution returns empty result set resulting in the exception. Both
> test failures are due to this exception. I tried debugging the query
> execution but didn't really go nowhere with it. I suspect this could be due
> to recent metastore changes. I tried reproducing this outside test but with
> no success.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19206) Automatic memory management for open streaming writers
[ https://issues.apache.org/jira/browse/HIVE-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19206: > Automatic memory management for open streaming writers > -- > > Key: HIVE-19206 > URL: https://issues.apache.org/jira/browse/HIVE-19206 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 > Environment: Problem: > When there are 100s of record updaters open, the amount of memory required by > orc writers keeps growing because of ORC's internal buffers. This can lead to > potential high GC or OOM during streaming ingest. > Solution: > The high level idea is for the streaming connection to remember all the open > record updaters and flush the record updater periodically (at some interval). > Records written to each record updater can be used as a metric to determine > the candidate record updaters for flushing. > If stripe size of orc file is 64MB, the default memory management check > happens only after every 5000 rows which may which may be too late when there > are too many concurrent writers in a process. Example case would be 100 > writers open and each of them have almost full stripe of 64MB buffered data, > this would take 100*64MB ~=6GB of memory. When all of the record writers > flush, the memory usage drops down to 100*~2MB which is just ~200MB memory > usage. >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19207) Support avro record writer for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19207: > Support avro record writer for streaming ingest > --- > > Key: HIVE-19207 > URL: https://issues.apache.org/jira/browse/HIVE-19207 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Add support for Avro record writer in streaming ingest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19208) Automatic distribution for non-bucketed tables during streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19208: > Automatic distribution for non-bucketed tables during streaming ingest > -- > > Key: HIVE-19208 > URL: https://issues.apache.org/jira/browse/HIVE-19208 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Currently non-bucketed tables write to a single file in transaction batch. > This limits the concurrency of compactor job to be 1. Provide an option to > automatically distribute the incoming records to N files within a transaction > batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19209) Streaming ingest record writers should accept input stream
[ https://issues.apache.org/jira/browse/HIVE-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19209: > Streaming ingest record writers should accept input stream > -- > > Key: HIVE-19209 > URL: https://issues.apache.org/jira/browse/HIVE-19209 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Record writers in streaming ingest currently accepts byte[]. Provide an > option for clients to pass in input stream directly from which byte[] for > record can be constructed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19210) Create separate module for streaming ingest
[ https://issues.apache.org/jira/browse/HIVE-19210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19210: > Create separate module for streaming ingest > --- > > Key: HIVE-19210 > URL: https://issues.apache.org/jira/browse/HIVE-19210 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This will retain the old hcat streaming API for old clients. The new > streaming ingest API will be separate module under hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19162) SMB : Test tez_smb_1.q stops making SMB join for a query
[ https://issues.apache.org/jira/browse/HIVE-19162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437737#comment-16437737 ] Deepak Jaiswal commented on HIVE-19162: --- Sure. These are all existing test cases. I will add new ones to junit and see if the existing ones can be moved too. Committed to master. > SMB : Test tez_smb_1.q stops making SMB join for a query > > > Key: HIVE-19162 > URL: https://issues.apache.org/jira/browse/HIVE-19162 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19162.1.patch > > > The test stopped making SMB join and instead creates a mapjoin. Likely a > change in stats issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19162) SMB : Test tez_smb_1.q stops making SMB join for a query
[ https://issues.apache.org/jira/browse/HIVE-19162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19162: -- Resolution: Fixed Status: Resolved (was: Patch Available) > SMB : Test tez_smb_1.q stops making SMB join for a query > > > Key: HIVE-19162 > URL: https://issues.apache.org/jira/browse/HIVE-19162 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19162.1.patch > > > The test stopped making SMB join and instead creates a mapjoin. Likely a > change in stats issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19211) New streaming ingest API and support for dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19211: > New streaming ingest API and support for dynamic partitioning > - > > Key: HIVE-19211 > URL: https://issues.apache.org/jira/browse/HIVE-19211 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > - New streaming API under new hive sub-module > - Dynamic partitioning support > - Delta file optimizations -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18964) HiveServer2 should not log errors when clients issue connection reset
[
https://issues.apache.org/jira/browse/HIVE-18964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437738#comment-16437738
]
Vihang Karajgaonkar commented on HIVE-18964:
Turns out we cannot fix this in Hive side because
{{TTransportFactory.getTransport}} does not throw a checked exception and hence
under-lying layers need to wrap the {{TTransportException}} in a
{{RuntimeException}} which is unnecessarily logged in {{TThreadpoolServer}}.
This issue needs to be fixed on Thrift side only. I spent a lot time to see if
we can configure HAProxy such that it issues a EOF during health-check but I
don't think there is a way to configure that. HAproxy issues a socket reset
once the tcp transport is opened which causes a TTransportException in SASL
negotiation phase.
> HiveServer2 should not log errors when clients issue connection reset
> -
>
> Key: HIVE-18964
> URL: https://issues.apache.org/jira/browse/HIVE-18964
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> HiveServer2 logs an ugly exception trace when clients issue a connection
> reset. There is nothing we can do when the connection is reset. This
> exception trace should either be ignored or logged only in debug mode.
> Things become worse if you are using load balancer like HAProxy which has its
> own health checks. HAProxy issues a connection reset to "quickly" close the
> connection once it finds that HS2 is up and available. This spams the logs at
> a very high frequency and makes them unusable for debugging purposes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19009) Retain and use runtime statistics during hs2 lifetime
[
https://issues.apache.org/jira/browse/HIVE-19009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437746#comment-16437746
]
Hive QA commented on HIVE-19009:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918763/HIVE-19009.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 158 failed/errored test(s), 13799 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,check_constraint_subquery.q,load_wrong_fileformat.q,check_constraint_udtf.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,check_constraint_nonboolean_expr.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,udf_sort_array_by_wrong2
[jira] [Updated] (HIVE-19206) Automatic memory management for open streaming writers
[ https://issues.apache.org/jira/browse/HIVE-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-19206: Environment: (was: Problem: When there are 100s of record updaters open, the amount of memory required by orc writers keeps growing because of ORC's internal buffers. This can lead to potential high GC or OOM during streaming ingest. Solution: The high level idea is for the streaming connection to remember all the open record updaters and flush the record updater periodically (at some interval). Records written to each record updater can be used as a metric to determine the candidate record updaters for flushing. If stripe size of orc file is 64MB, the default memory management check happens only after every 5000 rows which may which may be too late when there are too many concurrent writers in a process. Example case would be 100 writers open and each of them have almost full stripe of 64MB buffered data, this would take 100*64MB ~=6GB of memory. When all of the record writers flush, the memory usage drops down to 100*~2MB which is just ~200MB memory usage. ) > Automatic memory management for open streaming writers > -- > > Key: HIVE-19206 > URL: https://issues.apache.org/jira/browse/HIVE-19206 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19206) Automatic memory management for open streaming writers
[ https://issues.apache.org/jira/browse/HIVE-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-19206: Problem: When there are 100s of record updaters open, the amount of memory required by orc writers keeps growing because of ORC's internal buffers. This can lead to potential high GC or OOM during streaming ingest. Solution: The high level idea is for the streaming connection to remember all the open record updaters and flush the record updater periodically (at some interval). Records written to each record updater can be used as a metric to determine the candidate record updaters for flushing. If stripe size of orc file is 64MB, the default memory management check happens only after every 5000 rows which may which may be too late when there are too many concurrent writers in a process. Example case would be 100 writers open and each of them have almost full stripe of 64MB buffered data, this would take 100*64MB ~=6GB of memory. When all of the record writers flush, the memory usage drops down to 100*~2MB which is just ~200MB memory usage. > Automatic memory management for open streaming writers > -- > > Key: HIVE-19206 > URL: https://issues.apache.org/jira/browse/HIVE-19206 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 > Environment: Problem: > When there are 100s of record updaters open, the amount of memory required by > orc writers keeps growing because of ORC's internal buffers. This can lead to > potential high GC or OOM during streaming ingest. > Solution: > The high level idea is for the streaming connection to remember all the open > record updaters and flush the record updater periodically (at some interval). > Records written to each record updater can be used as a metric to determine > the candidate record updaters for flushing. > If stripe size of orc file is 64MB, the default memory management check > happens only after every 5000 rows which may which may be too late when there > are too many concurrent writers in a process. Example case would be 100 > writers open and each of them have almost full stripe of 64MB buffered data, > this would take 100*64MB ~=6GB of memory. When all of the record writers > flush, the memory usage drops down to 100*~2MB which is just ~200MB memory > usage. >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19206) Automatic memory management for open streaming writers
[ https://issues.apache.org/jira/browse/HIVE-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-19206: Description: Problem: When there are 100s of record updaters open, the amount of memory required by orc writers keeps growing because of ORC's internal buffers. This can lead to potential high GC or OOM during streaming ingest. Solution: The high level idea is for the streaming connection to remember all the open record updaters and flush the record updater periodically (at some interval). Records written to each record updater can be used as a metric to determine the candidate record updaters for flushing. If stripe size of orc file is 64MB, the default memory management check happens only after every 5000 rows which may which may be too late when there are too many concurrent writers in a process. Example case would be 100 writers open and each of them have almost full stripe of 64MB buffered data, this would take 100*64MB ~=6GB of memory. When all of the record writers flush, the memory usage drops down to 100*~2MB which is just ~200MB memory usage. > Automatic memory management for open streaming writers > -- > > Key: HIVE-19206 > URL: https://issues.apache.org/jira/browse/HIVE-19206 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Problem: > When there are 100s of record updaters open, the amount of memory required > by orc writers keeps growing because of ORC's internal buffers. This can lead > to potential high GC or OOM during streaming ingest. > Solution: > The high level idea is for the streaming connection to remember all the open > record updaters and flush the record updater periodically (at some interval). > Records written to each record updater can be used as a metric to determine > the candidate record updaters for flushing. > If stripe size of orc file is 64MB, the default memory management check > happens only after every 5000 rows which may which may be too late when there > are too many concurrent writers in a process. Example case would be 100 > writers open and each of them have almost full stripe of 64MB buffered data, > this would take 100*64MB ~=6GB of memory. When all of the record writers > flush, the memory usage drops down to 100*~2MB which is just ~200MB memory > usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-19206) Automatic memory management for open streaming writers
[ https://issues.apache.org/jira/browse/HIVE-19206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-19206: Comment: was deleted (was: Problem: When there are 100s of record updaters open, the amount of memory required by orc writers keeps growing because of ORC's internal buffers. This can lead to potential high GC or OOM during streaming ingest. Solution: The high level idea is for the streaming connection to remember all the open record updaters and flush the record updater periodically (at some interval). Records written to each record updater can be used as a metric to determine the candidate record updaters for flushing. If stripe size of orc file is 64MB, the default memory management check happens only after every 5000 rows which may which may be too late when there are too many concurrent writers in a process. Example case would be 100 writers open and each of them have almost full stripe of 64MB buffered data, this would take 100*64MB ~=6GB of memory. When all of the record writers flush, the memory usage drops down to 100*~2MB which is just ~200MB memory usage.) > Automatic memory management for open streaming writers > -- > > Key: HIVE-19206 > URL: https://issues.apache.org/jira/browse/HIVE-19206 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Problem: > When there are 100s of record updaters open, the amount of memory required > by orc writers keeps growing because of ORC's internal buffers. This can lead > to potential high GC or OOM during streaming ingest. > Solution: > The high level idea is for the streaming connection to remember all the open > record updaters and flush the record updater periodically (at some interval). > Records written to each record updater can be used as a metric to determine > the candidate record updaters for flushing. > If stripe size of orc file is 64MB, the default memory management check > happens only after every 5000 rows which may which may be too late when there > are too many concurrent writers in a process. Example case would be 100 > writers open and each of them have almost full stripe of 64MB buffered data, > this would take 100*64MB ~=6GB of memory. When all of the record writers > flush, the memory usage drops down to 100*~2MB which is just ~200MB memory > usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19205: - Description: This is umbrella jira to track hive streaming ingest improvements. At a high level following are the improvements - Support for dynamic partitioning - API changes (simple streaming connection builder) - Hide the transaction batches from clients (client can tune the transaction batch but doesn't have to know about the transaction batch size) - Support auto rollover to next transaction batch (clients don't have to worry about closing a transaction batch and opening a new one) - Record writers will all be strict meaning the schema of the record has to match table schema. This is to avoid the multiple serialization/deserialization for re-ordering columns if there is schema mismatch - Automatic distribution for non-bucketed tables so that compactor can have more parallelism - Create delta files with all ORC overhead disabled (no index, no compression, no dictionary). Compactor will recreate the orc files with compression and dictionary encoding. - Automatic memory management via auto-flushing (will yield smaller stripes for delta files but is more scalable and clients don't have to worry about distributing the data across writers) - Support for more writers (Avro specifically. ORC passthrough format?) - Support to accept input stream instead of record byte[] - Removing HCatalog dependency (old streaming API will be in the hcatalog package for backward compatibility, new streaming API will be in its own hive module) was: This is umbrella jira to track hive streaming ingest improvements. At a high level following are the improvements - Support for dynamic partitioning - API changes (simple streaming connection builder) - Hide the transaction batches from clients (client can tune the transaction batch but doesn't have to know about the transaction batch size) - Support auto rollover to next transaction batch (clients don't have to worry about closing a transaction batch and opening a new one) - Record writers will all be strict meaning the schema of the record has to match table schema. This is to avoid the multiple serialization/deserialization for re-ordering columns if there is schema mismatch - Automatic distribution for non-bucketed tables so that compactor can have more parallelism - Create delta files with all ORC overhead disabled (no compression, no dictionary). Compactor will recreate the orc files with compression and dictionary encoding. - Automatic memory management via auto-flushing (will yield smaller stripes for delta files but is more scalable and clients don't have to worry about distributing the data across writers) - Support for more writers (Avro specifically. ORC passthrough format?) - Support to accept input stream instead of record byte[] - Removing HCatalog dependency (old streaming API will be in the hcatalog package for backward compatibility, new streaming API will be in its own hive module) > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no index, no > compression, no dictionary). Compactor will recreate the orc files with > compression and dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog > package for backward com
[jira] [Updated] (HIVE-19205) Hive streaming ingest improvements (v2)
[ https://issues.apache.org/jira/browse/HIVE-19205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19205: - Description: This is umbrella jira to track hive streaming ingest improvements. At a high level following are the improvements - Support for dynamic partitioning - API changes (simple streaming connection builder) - Hide the transaction batches from clients (client can tune the transaction batch but doesn't have to know about the transaction batch size) - Support auto rollover to next transaction batch (clients don't have to worry about closing a transaction batch and opening a new one) - Record writers will all be strict meaning the schema of the record has to match table schema. This is to avoid the multiple serialization/deserialization for re-ordering columns if there is schema mismatch - Automatic distribution for non-bucketed tables so that compactor can have more parallelism - Create delta files with all ORC overhead disabled (no index, no compression, no dictionary). Compactor will recreate the orc files with index, compression and dictionary encoding. - Automatic memory management via auto-flushing (will yield smaller stripes for delta files but is more scalable and clients don't have to worry about distributing the data across writers) - Support for more writers (Avro specifically. ORC passthrough format?) - Support to accept input stream instead of record byte[] - Removing HCatalog dependency (old streaming API will be in the hcatalog package for backward compatibility, new streaming API will be in its own hive module) was: This is umbrella jira to track hive streaming ingest improvements. At a high level following are the improvements - Support for dynamic partitioning - API changes (simple streaming connection builder) - Hide the transaction batches from clients (client can tune the transaction batch but doesn't have to know about the transaction batch size) - Support auto rollover to next transaction batch (clients don't have to worry about closing a transaction batch and opening a new one) - Record writers will all be strict meaning the schema of the record has to match table schema. This is to avoid the multiple serialization/deserialization for re-ordering columns if there is schema mismatch - Automatic distribution for non-bucketed tables so that compactor can have more parallelism - Create delta files with all ORC overhead disabled (no index, no compression, no dictionary). Compactor will recreate the orc files with compression and dictionary encoding. - Automatic memory management via auto-flushing (will yield smaller stripes for delta files but is more scalable and clients don't have to worry about distributing the data across writers) - Support for more writers (Avro specifically. ORC passthrough format?) - Support to accept input stream instead of record byte[] - Removing HCatalog dependency (old streaming API will be in the hcatalog package for backward compatibility, new streaming API will be in its own hive module) > Hive streaming ingest improvements (v2) > --- > > Key: HIVE-19205 > URL: https://issues.apache.org/jira/browse/HIVE-19205 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > This is umbrella jira to track hive streaming ingest improvements. At a high > level following are the improvements > - Support for dynamic partitioning > - API changes (simple streaming connection builder) > - Hide the transaction batches from clients (client can tune the transaction > batch but doesn't have to know about the transaction batch size) > - Support auto rollover to next transaction batch (clients don't have to > worry about closing a transaction batch and opening a new one) > - Record writers will all be strict meaning the schema of the record has to > match table schema. This is to avoid the multiple > serialization/deserialization for re-ordering columns if there is schema > mismatch > - Automatic distribution for non-bucketed tables so that compactor can have > more parallelism > - Create delta files with all ORC overhead disabled (no index, no > compression, no dictionary). Compactor will recreate the orc files with > index, compression and dictionary encoding. > - Automatic memory management via auto-flushing (will yield smaller stripes > for delta files but is more scalable and clients don't have to worry about > distributing the data across writers) > - Support for more writers (Avro specifically. ORC passthrough format?) > - Support to accept input stream instead of record byte[] > - Removing HCatalog dependency (old streaming API will be in the hcatalog >
[jira] [Commented] (HIVE-19104) When test MetaStore is started with retry the instances should be independent
[
https://issues.apache.org/jira/browse/HIVE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437780#comment-16437780
]
Hive QA commented on HIVE-19104:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore: The patch generated 4 new + 266
unchanged - 2 fixed = 270 total (was 268) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10189/dev-support/hive-personality.sh
|
| git revision | master / 3915980 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10189/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: hcatalog/core itests/util standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10189/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> When test MetaStore is started with retry the instances should be independent
> -
>
> Key: HIVE-19104
> URL: https://issues.apache.org/jira/browse/HIVE-19104
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-19104.2.patch, HIVE-19104.3.patch,
> HIVE-19104.4.patch, HIVE-19104.5.patch, HIVE-19104.patch
>
>
> When multiple MetaStore instances are started with
> {{MetaStoreTestUtils.startMetaStoreWithRetry}} currently they use the same
> JDBC url, and warehouse directory. This can cause problem in the tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19077) Handle duplicate ptests requests standing in queue at the same time
[
https://issues.apache.org/jira/browse/HIVE-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437783#comment-16437783
]
Vihang Karajgaonkar commented on HIVE-19077:
I think instead of not running the build we should remove the already scheduled
build since ptests always uses the latest version of the attached patch. The
problem with this approach is though dev can "reserve" their spot in the queue
before the patch is ready and then work on the patch so that it is gets tested
sooner. [~djaiswal] just to understand your issue is the problem happening
because you attach a higher version of patch after rebasing? What happens if
you re-attach the patch with the same version number since that seems to be
what you intend to do, right? That way it will not be added to the queue again
right?
> Handle duplicate ptests requests standing in queue at the same time
> ---
>
> Key: HIVE-19077
> URL: https://issues.apache.org/jira/browse/HIVE-19077
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19077.0.patch, HIVE-19077.1.patch,
> HIVE-19077.sslFix.patch
>
>
> I've been keeping on eye on our {{PreCommit-HIVE-Build}} job, and what I
> noticed that sometimes huge queues can build up, that contain jira's more
> than once. (Yesterday I've seen a queue of 40, having 31 distinct jiras..)
> Simple scenario is that I upload a patch, it gets queued for ptest (already
> long queue), and 3 hours later I will update it, re-upload and re-queue. Now
> the current ptest infra seems to be smart enough to always deal with the
> latest patch, so what will happen is that the same patch will be tested 2
> times (with ~3 hours) diff, most probably with same result.
> I propose we do some deduplication - if ptest starts running the request for
> Jira X, then it can take a look on the current queue, and see if X is there
> again. If so, it can skip for now, it will be picked up later anyway.
> In practice this means that if you reconsider your patch and update it, your
> original place in the queue will be gone (like as a penalty for changing it),
> but overall it saves resources for the whole community.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437784#comment-16437784
]
Sahil Takiar commented on HIVE-18883:
-
Ok think I found the issue. This patch isn't working on ptest because the
{{patchprocess}} directory gets deleted as soon as Yetus starts to run because
it requires a clean workspace (we could specify the {{--dirty-workspace}}
option, but that seems dangerous). Yetus checks for the Findbugs directories
after it cleans the workspace, so it decides not to run Findbugs.
Looking at other Apache components, they install Findbugs in a docker image and
then set {{FINDBUGS_HOME}}. Unfortunately, we don't follow that model, so I
need to brainstorm an alternative place to install the Findbugs distribution.
It doesn't look like we can install it anywhere in the cloned git repo.
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19212) Fix findbugs yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-19212: --- > Fix findbugs yetus pre-commit checks > > > Key: HIVE-19212 > URL: https://issues.apache.org/jira/browse/HIVE-19212 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > Follow up from HIVE-18883, the committed patch isn't working and Findbugs is > still not working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved HIVE-18883.
-
Resolution: Fixed
Ok, think I find a way, we will just have to install the findbugs package in
the {{yetus-exec.sh}} script. I'll open another JIRA to do this.
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19212) Fix findbugs yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19212: Issue Type: Sub-task (was: Bug) Parent: HIVE-13503 > Fix findbugs yetus pre-commit checks > > > Key: HIVE-19212 > URL: https://issues.apache.org/jira/browse/HIVE-19212 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > Follow up from HIVE-18883, the committed patch isn't working and Findbugs is > still not working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19193) TestActivePassiveHA fails
[
https://issues.apache.org/jira/browse/HIVE-19193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437827#comment-16437827
]
Ashutosh Chauhan commented on HIVE-19193:
-
+1
> TestActivePassiveHA fails
> -
>
> Key: HIVE-19193
> URL: https://issues.apache.org/jira/browse/HIVE-19193
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Ashutosh Chauhan
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19193.1.patch
>
>
> This looks like a flaky test. If its not reproducible locally try improving
> on stability of this test, since it does fail randomly with Hive QA.
> {code:java}
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover(TestActivePassiveHA.java:356)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19194) TestDruidStorageHandler fails
[
https://issues.apache.org/jira/browse/HIVE-19194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437837#comment-16437837
]
Ashutosh Chauhan commented on HIVE-19194:
-
Builds which had this failure is now gone. If there is some way to improve
stability of this test (based on your guess of cause of ficklenss) then lets do
that else we close this ticket until we see the failure again.
> TestDruidStorageHandler fails
> -
>
> Key: HIVE-19194
> URL: https://issues.apache.org/jira/browse/HIVE-19194
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: Ashutosh Chauhan
>Assignee: slim bouguerra
>Priority: Major
>
> This tests fails randomly. If its not reproducible locally consider improving
> its stability since it does fail once in a while on Hive QA.
> {code}
> java.lang.AssertionError: expected:<0> but was:<1> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:743) at
> org.junit.Assert.assertEquals(Assert.java:118) at
> org.junit.Assert.assertEquals(Assert.java:555) at
> org.junit.Assert.assertEquals(Assert.java:542) at
> org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable(TestDruidStorageHandler.java:414)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
