[jira] [Commented] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[
https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351303#comment-16351303
]
Hive QA commented on HIVE-18617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909074/HIVE-18617.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.testutils.TestHiveTestEnvSetup.org.apache.hive.testutils.TestHiveTestEnvSetup
(batchId=295)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8997/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8997/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8997/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909074 - PreCommit-HIVE-Build
> Workload management Action parser does not generate the correct pool path.
> --
>
> Key: HIVE-18617
> URL: https://issues.apache.org/jira/browse/HIVE-18617
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-18617.01.patch
>
>
> For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but
> expected plan_2.pool
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351292#comment-16351292 ] Prasanth Jayachandran commented on HIVE-18617: -- +1 > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18617.01.patch > > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351290#comment-16351290
]
Prasanth Jayachandran commented on HIVE-18597:
--
+1
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[
https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351284#comment-16351284
]
Hive QA commented on HIVE-18617:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 4a33ec8 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8997/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Workload management Action parser does not generate the correct pool path.
> --
>
> Key: HIVE-18617
> URL: https://issues.apache.org/jira/browse/HIVE-18617
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-18617.01.patch
>
>
> For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but
> expected plan_2.pool
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351274#comment-16351274
]
Hive QA commented on HIVE-18597:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909071/HIVE-18597.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8996/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8996/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8996/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909071 - PreCommit-HIVE-Build
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351259#comment-16351259
]
Hive QA commented on HIVE-18597:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 4a33ec8 |
| Default Java | 1.8.0_111 |
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8996/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[
https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351253#comment-16351253
]
Hive QA commented on HIVE-18281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909070/HIVE-18281.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 12511 tests
executed
*Failed tests:*
{noformat}
TestAddAlterDropIndexes - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=206)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
TestGetListIndexes - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
TestTablesGetExists - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
TestTablesList - did not produce a TEST-*.xml file (likely timed out)
(batchId=206)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8995/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8995/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8995/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909070 - PreCommit-HIVE-Build
> HiveServer2 HA for LLAP and Workload Manager
>
>
> Key: HIVE-18281
> URL: https://issues.apache.org/jira/browse/HIVE-18281
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18281.1.patch, HIVE-18281.WIP.patch, HSI-HA.pdf
>
>
> When running HS2 with LLAP and Workload Manager, HS2 becomes single point of
> failure as some of the states for workload management and scheduling are
> maintained in-memory.
> The proposal is to support Active/Passive mode of high availability in which,
> all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen
> which will maintain stateful information. Clients using service discovery
> will always connect to the leader for submitting queries. There will also be
> some responsibilities for the leader, failover handling, tez session
> reconnect etc. Will
[jira] [Updated] (HIVE-18410) [Performance][Avro] Reading flat Avro tables is very expensive in Hive
[
https://issues.apache.org/jira/browse/HIVE-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ratandeep Ratti updated HIVE-18410:
---
Attachment: HIVE-18410_3.patch
> [Performance][Avro] Reading flat Avro tables is very expensive in Hive
> --
>
> Key: HIVE-18410
> URL: https://issues.apache.org/jira/browse/HIVE-18410
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.3.2
>Reporter: Ratandeep Ratti
>Assignee: Ratandeep Ratti
>Priority: Major
> Fix For: 3.0.0, 2.3.2
>
> Attachments: HIVE-18410.patch, HIVE-18410_1.patch,
> HIVE-18410_2.patch, HIVE-18410_3.patch, profiling_with_patch.nps,
> profiling_with_patch.png, profiling_without_patch.nps,
> profiling_without_patch.png
>
>
> There's a performance penalty when reading flat [no nested fields] Avro
> tables. When reading the same flat dataset in Pig, it takes half the time.
> On profiling, a lot of time is spent in
> {{AvroDeserializer.deserializeSingleItemNullableUnion()}}. The bulk of the
> time is spent in GenericData.get().resolveUnion(), which calls
> GenericData.getSchemaName(Object datum), which does a lot of instanceof
> checks. This could be simplified with performance benefits. A approach is
> described in this patch which almost halves the runtime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[
https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351244#comment-16351244
]
Hive QA commented on HIVE-18281:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} llap-common: The patch generated 1 new + 1 unchanged -
0 fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-client: The patch generated 33 new + 75 unchanged
- 8 fixed = 108 total (was 83) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-tez: The patch generated 6 new + 66 unchanged - 0
fixed = 72 total (was 66) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 4 new + 47 unchanged - 1 fixed
= 51 total (was 48) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 41 new + 18 unchanged - 2
fixed = 59 total (was 20) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-ext-client: The patch generated 1 new + 39
unchanged - 0 fixed = 40 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 0
unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} llap-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} llap-client generated 0 new + 6 unchanged - 6 fixed
= 6 total (was 12) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} llap-tez in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} llap-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} llap-ext-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} util in the patch passed.
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351233#comment-16351233
]
Hive QA commented on HIVE-18350:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909069/HIVE-18350.12.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 12966 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.org.apache.hadoop.hive.cli.TestContribNegativeCliDriver
(batchId=245)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestReplChangeManager.testRecycleNonPartTable
(batchId=223)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testTableOps
(batchId=213)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeCRLFOffInDSVOutput
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=231)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testMulti (batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchAbort
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchCommitPartitioned
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchCommitUnpartitioned
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyAbortPartitioned
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyAbortUnartitioned
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyCommitPartitioned
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testTransactionBatchEmptyCommitUnpartitioned
(batchId=202)
org.apache.hive.hcatalog.streaming.mutate.TestMutations.testUpdatesAndDeletes
(batchId=202)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveAndKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8994/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8994/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8994/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 40 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909069 - PreCommit-HIVE-Build
> load data should rename files consistent with
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351228#comment-16351228
]
Hive QA commented on HIVE-18350:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 6 new + 204 unchanged - 2
fixed = 210 total (was 206) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
58s{color} | {color:red} root: The patch generated 6 new + 597 unchanged - 2
fixed = 603 total (was 599) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 4a33ec8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8994/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8994/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8994/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8994/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch,
> HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.2.patch,
> HIVE-18350.3.patch, HIVE-18350.4.patch, HIVE-18350.5.patch,
> HIVE-18350.6.patch, HIVE-18350.7.patch, HIVE-18350.8.patch, HIVE-18350.9.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios and may
> cause trouble for other types of queries in future.
> We
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351216#comment-16351216
]
Vihang Karajgaonkar commented on HIVE-18421:
adding reviewboard link
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18421:
---
Status: Patch Available (was: Open)
[~mmccline] [~gopalv] [~aihuaxu] Can you please take a look?
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.3.2, 2.2.0, 2.1.1, 3.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351212#comment-16351212
]
Vihang Karajgaonkar commented on HIVE-18421:
Updated the Jira description appropriately. This patch doesn't add code to
handle overflows in a "right" manner. It only makes vectorized code execution
handle overflow in a "consistent" manner when compared to non-vectorized
execution.
I feel that handling overflows according to a configurable policy (wraparound,
warn, error, set to null) would be a good addition and it can be taken up as a
separate effort.
Attaching the first version of this patch. I reviewed almost all the vector
expressions (phew ..;)) and found a subset of expressions which do not handle
overflows in a consistent manner. As discussed above, it introduces a new
config which enables usage of checked expressions when available.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18421:
---
Attachment: HIVE-18421.01.patch
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for the review Vihang.
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18421:
---
Summary: Vectorized execution handles overflows in a different manner than
non-vectorized execution (was: Vectorized execution does not handle integer
overflows)
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18616) work around HADOOP-15171 p2
[
https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351206#comment-16351206
]
Hive QA commented on HIVE-18616:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909059/HIVE-18616.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 12969 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.templeton.TestConcurrentJobRequestsThreadsAndTimeout.ConcurrentListJobsVerifyExceptions
(batchId=191)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8993/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8993/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8993/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909059 - PreCommit-HIVE-Build
> work around HADOOP-15171 p2
> ---
>
> Key: HIVE-18616
> URL: https://issues.apache.org/jira/browse/HIVE-18616
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18616.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16392) Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance logic
[
https://issues.apache.org/jira/browse/HIVE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351200#comment-16351200
]
Sergey Shelukhin commented on HIVE-16392:
-
Hmm... this makes permission management more complex for ppl still using FS
based permissions. Is there particular reason to remove this? We could just
have it false by default. cc [~ashutoshc]
> Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance
> logic
>
>
> Key: HIVE-16392
> URL: https://issues.apache.org/jira/browse/HIVE-16392
> Project: Hive
> Issue Type: Task
> Components: Security
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Labels: backwards-incompatible
> Fix For: 3.0.0
>
> Attachments: HIVE-16392.1.patch, HIVE-16392.2.patch
>
>
> As discussed in HIVE-16346 we should remove the config
> {{hive.warehouse.subdir.inherit.perms}} and all the permissions inheritance
> logic.
> This feature is no longer needed in Hive as the traditional permission model
> has largely been replaced by external security systems such as Ranger and
> Sentry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18617: - Status: Patch Available (was: Open) Fixed bug in Action parser for pool path. [~prasanth_j] Please review. > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18617.01.patch > > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18617: - Attachment: HIVE-18617.01.patch > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18617.01.patch > > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-18617: > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18607: Attachment: HIVE-18607.02.patch > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.patch > > > I cannot get the HBaseCliDriver to run locally, so first I'll use HiveQA to > check smth. > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18616) work around HADOOP-15171 p2
[
https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351187#comment-16351187
]
Hive QA commented on HIVE-18616:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 111 unchanged - 1
fixed = 112 total (was 112) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 47f4570 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8993/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8993/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> work around HADOOP-15171 p2
> ---
>
> Key: HIVE-18616
> URL: https://issues.apache.org/jira/browse/HIVE-18616
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18616.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18597:
---
Attachment: HIVE-18597.1.patch
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18597:
---
Status: Patch Available (was: Open)
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-18597.1.patch
>
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18597) LLAP: Packaging for log4j NDC has two log4j jars it can pick
[
https://issues.apache.org/jira/browse/HIVE-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-18597:
--
Assignee: Gopal V
> LLAP: Packaging for log4j NDC has two log4j jars it can pick
>
>
> Key: HIVE-18597
> URL: https://issues.apache.org/jira/browse/HIVE-18597
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
>
> The log4j 1 and 2.x jars have NDC classes with identical package names.
> {code}
> // log4j-1.2-API needed for NDC
> org.apache.log4j.NDC.class,
> {code}
> Queries fail with
> {code}
> java.lang.ClassCastException: org.apache.log4j.NDC$DiagnosticContext cannot
> be cast to java.lang.String
> at
> org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.setMDCFromNDC(TaskRunnerCallable.java:312)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {code}
> If the service driver packages the log4j1.x jar instead of the 2.x API shim
> jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18281: - Status: Patch Available (was: Open) Added some tests. Pending items - Service discovery - Session reconnection - Add /peers web endpoint Will create separate jiras for these. [~sershe]/[~thejas] can someone please take a look? > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18281.1.patch, HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18281: - Attachment: HIVE-18281.1.patch > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18281.1.patch, HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351165#comment-16351165
]
Hive QA commented on HIVE-18350:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909051/HIVE-18350.11.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 12967 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testTableOps
(batchId=213)
org.apache.hadoop.hive.metastore.client.TestAddPartitions.testAddPartitionsNullColTypeInSd[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestAppendPartitions.testAppendPartWrongColumnInPartName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaCaseSensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestGetTableMeta.testGetTableMetaNullOrEmptyDb[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8992/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8992/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8992/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909051 - PreCommit-HIVE-Build
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch,
> HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.2.patch,
> HIVE-18350.3.patch, HIVE-18350.4.patch, HIVE-18350.5.patch,
> HIVE-18350.6.patch, HIVE-18350.7.patch, HIVE-18350.8.patch, HIVE-18350.9.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios and may
> cause trouble for other types of queries in future.
> We need consistent naming convention.
> For non-bucketed table, hive renames all the files regardless of how they
> were named by the user.
> For bucketed table, hive relies on
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.12.patch > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.2.patch, > HIVE-18350.3.patch, HIVE-18350.4.patch, HIVE-18350.5.patch, > HIVE-18350.6.patch, HIVE-18350.7.patch, HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351146#comment-16351146
]
Hive QA commented on HIVE-18350:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 6 new + 204 unchanged - 2
fixed = 210 total (was 206) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
56s{color} | {color:red} root: The patch generated 6 new + 597 unchanged - 2
fixed = 603 total (was 599) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / f9efd84 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8992/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8992/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8992/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8992/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch,
> HIVE-18350.11.patch, HIVE-18350.2.patch, HIVE-18350.3.patch,
> HIVE-18350.4.patch, HIVE-18350.5.patch, HIVE-18350.6.patch,
> HIVE-18350.7.patch, HIVE-18350.8.patch, HIVE-18350.9.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios and may
> cause trouble for other types of queries in future.
> We need consistent naming
[jira] [Assigned] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-18599:
--
Assignee: Steve Yeom (was: Gopal V)
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch, HIVE-18599.04.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18599:
---
Attachment: HIVE-18599.04.patch
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch, HIVE-18599.04.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-18599:
--
Assignee: Gopal V (was: Steve Yeom)
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch, HIVE-18599.04.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18599:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~steveyeom2017]
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.11.patch > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.11.patch, HIVE-18350.2.patch, HIVE-18350.3.patch, > HIVE-18350.4.patch, HIVE-18350.5.patch, HIVE-18350.6.patch, > HIVE-18350.7.patch, HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: (was: HIVE-18350.11.patch) > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.2.patch, HIVE-18350.3.patch, HIVE-18350.4.patch, > HIVE-18350.5.patch, HIVE-18350.6.patch, HIVE-18350.7.patch, > HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master thanks Gopal for the review > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18589.01.patch, HIVE-18589.02.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18616) work around HADOOP-15171 p2
[ https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351055#comment-16351055 ] Gopal V commented on HIVE-18616: LGTM - +1 Added to the weekend tests, I'll do a heapdump and check if there are any unreferenced codecs in the heap. > work around HADOOP-15171 p2 > --- > > Key: HIVE-18616 > URL: https://issues.apache.org/jira/browse/HIVE-18616 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18616.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18616) work around HADOOP-15171 p2
[ https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18616: Status: Patch Available (was: Open) This removes the copy fix, instead calling reset on codec every time. Also maintains the 0-byte check before it's fixed in the direct decompressor. [~gopalv] can you take a look? > work around HADOOP-15171 p2 > --- > > Key: HIVE-18616 > URL: https://issues.apache.org/jira/browse/HIVE-18616 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18616.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18616) work around HADOOP-15171 p2
[ https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18616: Attachment: HIVE-18616.patch > work around HADOOP-15171 p2 > --- > > Key: HIVE-18616 > URL: https://issues.apache.org/jira/browse/HIVE-18616 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18616.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18616) work around HADOOP-15171 p2
[ https://issues.apache.org/jira/browse/HIVE-18616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18616: --- > work around HADOOP-15171 p2 > --- > > Key: HIVE-18616 > URL: https://issues.apache.org/jira/browse/HIVE-18616 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18599) Transactions: Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18599:
---
Summary: Transactions: Fix CTAS on Micromanaged tables (was: Fix CTAS on
Micromanaged tables)
> Transactions: Fix CTAS on Micromanaged tables
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18599) Fix CTAS on Micromanaged tables
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-18599:
---
Summary: Fix CTAS on Micromanaged tables (was: CREATE TEMPORARY TABLE AS
SELECT(CTTAS) on Micromanaged table does not write data)
> Fix CTAS on Micromanaged tables
> ---
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.11.patch > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.11.patch, HIVE-18350.2.patch, HIVE-18350.3.patch, > HIVE-18350.4.patch, HIVE-18350.5.patch, HIVE-18350.6.patch, > HIVE-18350.7.patch, HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350984#comment-16350984 ] Gopal V commented on HIVE-18589: LGTM - +1 Not entirely happy with the metafile idea, but this can be revisited as HIVE-18320 makes more progress. > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch, HIVE-18589.02.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18125) Support arbitrary file names in input to Load Data
[ https://issues.apache.org/jira/browse/HIVE-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-18125. --- Resolution: Fixed Fix Version/s: 3.0.0 fixed via HIVE-18516 > Support arbitrary file names in input to Load Data > -- > > Key: HIVE-18125 > URL: https://issues.apache.org/jira/browse/HIVE-18125 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18125.01.patch, HIVE-18125.02.patch > > > HIVE-17361 only allows 0_0 and _0_copy_1. Should it support > arbitrary names? > If so, should it sort them and rename _0, 0001_0, etc? > This is probably a lot easier than changing the whole code base to assign > proper 'bucket' (writerId) everywhere Acid reads such file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Status: Open (was: Patch Available) > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, > HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, > HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, > HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch, > HIVE-18221.19.patch, HIVE-18221.20.patch, HIVE-18221.21.patch, > HIVE-18221.22.patch, HIVE-18221.23.patch, HIVE-18221.24.patch, > HIVE-18221.26.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~jdere] and [~ekoifman] for the reviews. > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.10.patch, > HIVE-18516.2.patch, HIVE-18516.3.patch, HIVE-18516.4.patch, > HIVE-18516.5.patch, HIVE-18516.6.patch, HIVE-18516.7.patch, > HIVE-18516.8.patch, HIVE-18516.9.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Release Note: N/A
Status: Resolved (was: Patch Available)
pushed to master
Thanks Sergey for the review
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch,
> HIVE-18606.03.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350945#comment-16350945 ] Eugene Koifman commented on HIVE-18516: --- +1 > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.10.patch, > HIVE-18516.2.patch, HIVE-18516.3.patch, HIVE-18516.4.patch, > HIVE-18516.5.patch, HIVE-18516.6.patch, HIVE-18516.7.patch, > HIVE-18516.8.patch, HIVE-18516.9.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18599) CREATE TEMPORARY TABLE AS SELECT(CTTAS) on Micromanaged table does not write data
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350928#comment-16350928
]
Gopal V commented on HIVE-18599:
LGTM - +1
> CREATE TEMPORARY TABLE AS SELECT(CTTAS) on Micromanaged table does not write
> data
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris
[
https://issues.apache.org/jira/browse/HIVE-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350802#comment-16350802
]
Vihang Karajgaonkar edited comment on HIVE-18449 at 2/2/18 8:05 PM:
Patch merged in master branch. Hi [~janulatha] can you provide a patch for
branch-2 as well since this patch does not apply directly to branch-2 and has
some conflicts. Thanks!
Also, since this change introduces a new config we should update the wiki
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties
was (Author: vihangk1):
Patch merged in master branch. Hi [~janulatha] can you provide a patch for
branch-2 as well since this patch does not apply directly to branch-2 and has
some conflicts. Thanks!
> Add configurable policy for choosing the HMS URI from hive.metastore.uris
> -
>
> Key: HIVE-18449
> URL: https://issues.apache.org/jira/browse/HIVE-18449
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18449.1.patch, HIVE-18449.2.patch
>
>
> HIVE-10815 added logic to randomly choose a HMS URI from
> {{hive.metastore.uris}}. It would be nice if there was a configurable policy
> that determined how a URI is chosen from this list - e.g. one option can be
> to randomly pick a URI, another option can be to choose the first URI in the
> list (which was the behavior prior to HIVE-10815).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350870#comment-16350870 ] Sergey Shelukhin commented on HIVE-18614: - Upper case identifiers without quotes are going to cause issues on Postgres due to the latter's bizarre SQL dialect rules. It's better to disable MySQL's bizarre SQL dialect rules by calling "SET @@session.sql_mode=ANSI_QUOTES" (see MetaStoreDirectSql.java for an example). Otherwise looks good. > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Attachment: HIVE-18606.03.patch
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch,
> HIVE-18606.03.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18518) Upgrade druid version to 0.11.0
[ https://issues.apache.org/jira/browse/HIVE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18518: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Nishant! > Upgrade druid version to 0.11.0 > --- > > Key: HIVE-18518 > URL: https://issues.apache.org/jira/browse/HIVE-18518 > Project: Hive > Issue Type: Bug >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18518.1.patch, HIVE-18518.patch > > > this task is to upgrade to druid version 0.11.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris
[
https://issues.apache.org/jira/browse/HIVE-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350802#comment-16350802
]
Vihang Karajgaonkar commented on HIVE-18449:
Patch merged in master branch. Hi [~janulatha] can you provide a patch for
branch-2 as well since this patch does not apply directly to branch-2 and has
some conflicts. Thanks!
> Add configurable policy for choosing the HMS URI from hive.metastore.uris
> -
>
> Key: HIVE-18449
> URL: https://issues.apache.org/jira/browse/HIVE-18449
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18449.1.patch, HIVE-18449.2.patch
>
>
> HIVE-10815 added logic to randomly choose a HMS URI from
> {{hive.metastore.uris}}. It would be nice if there was a configurable policy
> that determined how a URI is chosen from this list - e.g. one option can be
> to randomly pick a URI, another option can be to choose the first URI in the
> list (which was the behavior prior to HIVE-10815).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul
[ https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350788#comment-16350788 ] Vihang Karajgaonkar commented on HIVE-18347: Hi [~szehon] Yes you are right. HIVE-18449 introduces a different knob for the selection policy. You can ignore my review comment. LGTM +1 > Allow dynamic lookup of Hive Metastores via Consul > -- > > Key: HIVE-18347 > URL: https://issues.apache.org/jira/browse/HIVE-18347 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, > HIVE-18347.3.patch, HIVE-18347.4.patch > > > In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos > as dynamic services for scalability and flexibility. > In this architecture, we would like to allow HiveServer2 to dynamically load > balance between Metastores (which may be scaled up and down or to different > nodes) for different requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul
[ https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350780#comment-16350780 ] Szehon Ho commented on HIVE-18347: -- Hi Vihang, thanks for the review! Sorry about the late response. But about the comments, I was under the impression (at least from Thejas's comment about HIVE-18449) that we wanted to keep the selection policy as a separate knob of Hive under Hive control, and not via this hook, which was what the review comment is suggesting? I think HIVE-19449 seems to already give another knob for offering some predefined selection mechanism, so I think we shouldn't incorporate all that into a defaultHook that can be over-riden. What do you think? > Allow dynamic lookup of Hive Metastores via Consul > -- > > Key: HIVE-18347 > URL: https://issues.apache.org/jira/browse/HIVE-18347 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, > HIVE-18347.3.patch, HIVE-18347.4.patch > > > In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos > as dynamic services for scalability and flexibility. > In this architecture, we would like to allow HiveServer2 to dynamically load > balance between Metastores (which may be scaled up and down or to different > nodes) for different requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[
https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-18447:
Release Note:
Cookies can now be passed like:
{code}
jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
{code}
was:
Cookies can now be passed like:
{{jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=}}
> JDBC: Provide a way for JDBC users to pass cookie info via connection string
>
>
> Key: HIVE-18447
> URL: https://issues.apache.org/jira/browse/HIVE-18447
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch
>
>
> Some authentication mechanisms like Single Sign On, need the ability to pass
> a cookie to some intermediate authentication service like Knox via the JDBC
> driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP
> transport mode).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[
https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-18447:
Release Note:
Cookies can now be passed like:
{{jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=}}
was:
Cookies can now be passed like:
{code}
jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
{code}
> JDBC: Provide a way for JDBC users to pass cookie info via connection string
>
>
> Key: HIVE-18447
> URL: https://issues.apache.org/jira/browse/HIVE-18447
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch
>
>
> Some authentication mechanisms like Single Sign On, need the ability to pass
> a cookie to some intermediate authentication service like Knox via the JDBC
> driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP
> transport mode).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[
https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-18447:
Release Note:
Cookies can now be passed like:
{code}
jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
{code}
was:
Cookies can now be passed like:
jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.=
> JDBC: Provide a way for JDBC users to pass cookie info via connection string
>
>
> Key: HIVE-18447
> URL: https://issues.apache.org/jira/browse/HIVE-18447
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch
>
>
> Some authentication mechanisms like Single Sign On, need the ability to pass
> a cookie to some intermediate authentication service like Knox via the JDBC
> driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP
> transport mode).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[ https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18447: Release Note: Cookies can now be passed like: jdbc:hive2://:/;transportMode=http;httpPath=;http.cookie.=;http.cookie.= > JDBC: Provide a way for JDBC users to pass cookie info via connection string > > > Key: HIVE-18447 > URL: https://issues.apache.org/jira/browse/HIVE-18447 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch > > > Some authentication mechanisms like Single Sign On, need the ability to pass > a cookie to some intermediate authentication service like Knox via the JDBC > driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP > transport mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[ https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18447: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to master. Thanks [~thejas] > JDBC: Provide a way for JDBC users to pass cookie info via connection string > > > Key: HIVE-18447 > URL: https://issues.apache.org/jira/browse/HIVE-18447 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch > > > Some authentication mechanisms like Single Sign On, need the ability to pass > a cookie to some intermediate authentication service like Knox via the JDBC > driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP > transport mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18518) Upgrade druid version to 0.11.0
[ https://issues.apache.org/jira/browse/HIVE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350766#comment-16350766 ] Ashutosh Chauhan commented on HIVE-18518: - +1 > Upgrade druid version to 0.11.0 > --- > > Key: HIVE-18518 > URL: https://issues.apache.org/jira/browse/HIVE-18518 > Project: Hive > Issue Type: Bug >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-18518.1.patch, HIVE-18518.patch > > > this task is to upgrade to druid version 0.11.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[
https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350765#comment-16350765
]
Hive QA commented on HIVE-18513:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908906/HIVE-18513.5.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12973 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8991/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8991/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8991/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908906 - PreCommit-HIVE-Build
> Query results caching
> -
>
> Key: HIVE-18513
> URL: https://issues.apache.org/jira/browse/HIVE-18513
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch,
> HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch
>
>
> Add a query results cache that can save the results of an executed Hive query
> for reuse on subsequent queries. This may be useful in cases where the same
> query is issued many times, since Hive can return back the results of a
> cached query rather than having to execute the full query on the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[
https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350741#comment-16350741
]
Hive QA commented on HIVE-18513:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 18 new + 1328 unchanged - 2
fixed = 1346 total (was 1330) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
57s{color} | {color:red} root: The patch generated 18 new + 1881 unchanged - 2
fixed = 1899 total (was 1883) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 39f1e82 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8991/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8991/yetus/diff-checkstyle-root.txt
|
| modules | C: common ql service . itests itests/hive-blobstore itests/util U:
. |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8991/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query results caching
> -
>
> Key: HIVE-18513
> URL: https://issues.apache.org/jira/browse/HIVE-18513
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch,
> HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch
>
>
> Add a query results cache that can save the results of an executed Hive query
> for reuse on subsequent queries. This may be useful in cases where the same
> query is issued many times, since Hive can return back the results of a
> cached query rather than having to execute the full query on the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18532) Hive Connection can't be closed if there is an error while creating session
[
https://issues.apache.org/jira/browse/HIVE-18532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated HIVE-18532:
--
Attachment: HIVE-18532.1.patch
> Hive Connection can't be closed if there is an error while creating session
> ---
>
> Key: HIVE-18532
> URL: https://issues.apache.org/jira/browse/HIVE-18532
> Project: Hive
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Major
> Attachments: HIVE-18532.1.patch
>
>
> To open a hiveconnection we have to
> {code}
> HiveConnection connection = null;
> try {
> connection = new HiveConnection(uri, new Properties());
> } finally {
> if (connection != null)
> try {
> connection.close();
> } catch (SQLException e) {
> LOGGER.error("Error closing connection, ignored.", e);
> }
> }
> {code}
> HiveConnection opens the opensTransport and tries to create a session in the
> constructor.
> If opensTransport succeed, but session creates fails because of namenode
> error or any other error, HiveConnection can't be closed.
> Since construction fails, the hiveconnection object will be null and the
> client can't close it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18573: -- Description: Currently, Hive is mostly using user-defined black box sql operators during Query planning. It will be more beneficial to use proper calcite operators, this is to prepare the ground for pushing complex expressions to Druid-calcite adapter. was: Currently, Hive is mostly using user-defined black box sql operators during Query planning. It will be more beneficial to use proper calcite operators. Also, Use a single name for Extract operator instead of a different name for every Unit, Same for Floor function. This will allow unifying the treatment per operator. > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, > HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch, > HIVE-18573.7.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators, > this is to prepare the ground for pushing complex expressions to > Druid-calcite adapter. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18573: -- Attachment: HIVE-18573.7.patch > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, > HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch, > HIVE-18573.7.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators. > Also, Use a single name for Extract operator instead of a different name for > every Unit, > Same for Floor function. This will allow unifying the treatment per operator. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18532) Hive Connection can't be closed if there is an error while creating session
[
https://issues.apache.org/jira/browse/HIVE-18532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purshotam Shah updated HIVE-18532:
--
Attachment: (was: HIVE-18532.1.patch)
> Hive Connection can't be closed if there is an error while creating session
> ---
>
> Key: HIVE-18532
> URL: https://issues.apache.org/jira/browse/HIVE-18532
> Project: Hive
> Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
>Priority: Major
> Attachments: HIVE-18532.1.patch
>
>
> To open a hiveconnection we have to
> {code}
> HiveConnection connection = null;
> try {
> connection = new HiveConnection(uri, new Properties());
> } finally {
> if (connection != null)
> try {
> connection.close();
> } catch (SQLException e) {
> LOGGER.error("Error closing connection, ignored.", e);
> }
> }
> {code}
> HiveConnection opens the opensTransport and tries to create a session in the
> constructor.
> If opensTransport succeed, but session creates fails because of namenode
> error or any other error, HiveConnection can't be closed.
> Since construction fails, the hiveconnection object will be null and the
> client can't close it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18546) Remove unnecessary code introduced in HIVE-14498
[ https://issues.apache.org/jira/browse/HIVE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18546: --- Attachment: (was: HIVE-18546.05.patch) > Remove unnecessary code introduced in HIVE-14498 > > > Key: HIVE-18546 > URL: https://issues.apache.org/jira/browse/HIVE-18546 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18546.02.patch, HIVE-18546.03.patch, > HIVE-18546.04.patch, HIVE-18546.06.patch > > > HIVE-14498 introduced some code to check the invalidation of materialized > views that can be simplified, relying instead on existing transaction ids. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18546) Remove unnecessary code introduced in HIVE-14498
[ https://issues.apache.org/jira/browse/HIVE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18546: --- Attachment: HIVE-18546.06.patch > Remove unnecessary code introduced in HIVE-14498 > > > Key: HIVE-18546 > URL: https://issues.apache.org/jira/browse/HIVE-18546 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18546.02.patch, HIVE-18546.03.patch, > HIVE-18546.04.patch, HIVE-18546.06.patch > > > HIVE-14498 introduced some code to check the invalidation of materialized > views that can be simplified, relying instead on existing transaction ids. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18615) Prevent deserializing transaction list in the materializations invalidation cache multiple times
[ https://issues.apache.org/jira/browse/HIVE-18615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-18615: -- > Prevent deserializing transaction list in the materializations invalidation > cache multiple times > > > Key: HIVE-18615 > URL: https://issues.apache.org/jira/browse/HIVE-18615 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17627) Use druid scan query instead of the select query.
[ https://issues.apache.org/jira/browse/HIVE-17627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-17627: Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Use druid scan query instead of the select query. > - > > Key: HIVE-17627 > URL: https://issues.apache.org/jira/browse/HIVE-17627 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17627.1.patch, HIVE-17627.2.patch, HIVE-17627.patch > > > The biggest difference between select query and scan query is that, scan > query doesn't retain all rows in memory before rows can be returned to client. > It will cause memory pressure if too many rows required by select query. > Scan query doesn't have this issue. > Scan query can return all rows without issuing another pagination query, > which is extremely useful when query against historical or realtime node > directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18518) Upgrade druid version to 0.11.0
[ https://issues.apache.org/jira/browse/HIVE-18518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350698#comment-16350698 ] Nishant Bangarwa commented on HIVE-18518: - [~jcamachorodriguez] This need not wait for calcite update, can we merge this one ? > Upgrade druid version to 0.11.0 > --- > > Key: HIVE-18518 > URL: https://issues.apache.org/jira/browse/HIVE-18518 > Project: Hive > Issue Type: Bug >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-18518.1.patch, HIVE-18518.patch > > > this task is to upgrade to druid version 0.11.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17627) Use druid scan query instead of the select query.
[ https://issues.apache.org/jira/browse/HIVE-17627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-17627: Attachment: HIVE-17627.2.patch > Use druid scan query instead of the select query. > - > > Key: HIVE-17627 > URL: https://issues.apache.org/jira/browse/HIVE-17627 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17627.1.patch, HIVE-17627.2.patch, HIVE-17627.patch > > > The biggest difference between select query and scan query is that, scan > query doesn't retain all rows in memory before rows can be returned to client. > It will cause memory pressure if too many rows required by select query. > Scan query doesn't have this issue. > Scan query can return all rows without issuing another pagination query, > which is extremely useful when query against historical or realtime node > directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: HIVE-18614.patch > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18614 started by Miklos Gergely. - > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350655#comment-16350655
]
Hive QA commented on HIVE-18606:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908896/HIVE-18606.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12968 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCtasEmpty (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCtasEmpty (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.TestBeeLineWithArgs.testEscapeCRLFOffInDSVOutput
(batchId=231)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8990/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8990/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8990/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908896 - PreCommit-HIVE-Build
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
>
[jira] [Commented] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350654#comment-16350654 ] Xuefu Zhang commented on HIVE-18513: That's good. As a minor comment, if we were only doing time-based invalidation, there wouldn't be much a different between internal table and external table. > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch, > HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Patch Available (was: In Progress) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-18614: - > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18612) Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
[
https://issues.apache.org/jira/browse/HIVE-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-18612:
--
Status: Patch Available (was: In Progress)
> Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
> --
>
> Key: HIVE-18612
> URL: https://issues.apache.org/jira/browse/HIVE-18612
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18612.0.patch
>
>
> As per this jira comment made by Yetus maven plugins that want to use
> {{java}} executable are seeing a 1.7 java binary. In this particular case
> Yetus sets JAVA_HOME to a 1.8 JDK installation, and thus maven uses that, but
> any subsequent {{java}} executes will use the JRE which they see on PATH.
> This should be fixed by adding the proper java/bin (that of JAVA_HOME
> setting) to PATH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18613) Extend JsonSerDe to support BINARY type
[
https://issues.apache.org/jira/browse/HIVE-18613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-18613:
--
> Extend JsonSerDe to support BINARY type
> ---
>
> Key: HIVE-18613
> URL: https://issues.apache.org/jira/browse/HIVE-18613
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> It seems only type not supported currently, from the code:
> {code}
> case BINARY:
> throw new IOException("JsonSerDe does not support BINARY type");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18612) Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
[
https://issues.apache.org/jira/browse/HIVE-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-18612:
--
Attachment: HIVE-18612.0.patch
> Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
> --
>
> Key: HIVE-18612
> URL: https://issues.apache.org/jira/browse/HIVE-18612
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-18612.0.patch
>
>
> As per this jira comment made by Yetus maven plugins that want to use
> {{java}} executable are seeing a 1.7 java binary. In this particular case
> Yetus sets JAVA_HOME to a 1.8 JDK installation, and thus maven uses that, but
> any subsequent {{java}} executes will use the JRE which they see on PATH.
> This should be fixed by adding the proper java/bin (that of JAVA_HOME
> setting) to PATH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18599) CREATE TEMPORARY TABLE AS SELECT(CTTAS) on Micromanaged table does not write data
[
https://issues.apache.org/jira/browse/HIVE-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350618#comment-16350618
]
Steve Yeom commented on HIVE-18599:
---
Hi [~gopalv] Please see the last p-test results. There is no failures.
Also I have verified the 4 failed tests of the second last p-tests run
are OK by running those in my environment.
> CREATE TEMPORARY TABLE AS SELECT(CTTAS) on Micromanaged table does not write
> data
> -
>
> Key: HIVE-18599
> URL: https://issues.apache.org/jira/browse/HIVE-18599
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18599.01.patch, HIVE-18599.02.patch,
> HIVE-18599.03.patch
>
>
> CTTAS on temporary micromanaged table does not write data.
> I.e., "SELECT * FROM ctas0_mm;" does not return any rows from the below
> script:
>
> set hive.mapred.mode=nonstrict;
> set hive.explain.user=false;
> set hive.fetch.task.conversion=none;
> set tez.grouping.min-size=1;
> set tez.grouping.max-size=2;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
>
> drop table intermediate;
> create table intermediate(key int) partitioned by (p int) stored as orc;
> insert into table intermediate partition(p='455') select distinct key from
> src where key >= 0 order by key desc limit 2;
> insert into table intermediate partition(p='456') select distinct key from
> src where key is not null order by key asc limit 2;
> insert into table intermediate partition(p='457') select distinct key from
> src where key >= 100 order by key asc limit 2;
>
> drop table ctas0_mm;
> explain create temporary table ctas0_mm tblproperties
> ("transactional"="true", "transactional_properties"="insert_only") as select
> * from intermediate;
> create temporary table ctas0_mm tblproperties ("transactional"="true",
> "transactional_properties"="insert_only") as select * from intermediate;
>
> select * from ctas0_mm;
> drop table ctas0_mm;
> drop table intermediate;
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350583#comment-16350583
]
Hive QA commented on HIVE-18606:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
57s{color} | {color:red} ql: The patch generated 1 new + 322 unchanged - 0
fixed = 323 total (was 322) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 39f1e82 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8990/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8990/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at
[jira] [Assigned] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-17852: --- Assignee: Laszlo Bodor > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350539#comment-16350539
]
Hive QA commented on HIVE-18350:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908892/HIVE-18350.10.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 12966 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=248)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_7]
(batchId=59)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_7]
(batchId=133)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestReplChangeManager.testRecycleNonPartTable
(batchId=223)
org.apache.hadoop.hive.metastore.TestReplChangeManager.testRecyclePartTable
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hadoop.hive.ql.metadata.TestHive.testTable (batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHive.testThriftTable (batchId=278)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testTable (batchId=279)
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testThriftTable (batchId=279)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8989/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8989/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8989/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908892 - PreCommit-HIVE-Build
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch,
> HIVE-18350.2.patch, HIVE-18350.3.patch, HIVE-18350.4.patch,
> HIVE-18350.5.patch, HIVE-18350.6.patch, HIVE-18350.7.patch,
> HIVE-18350.8.patch, HIVE-18350.9.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350534#comment-16350534
]
Hive QA commented on HIVE-18350:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 6 new + 186 unchanged - 2
fixed = 192 total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
13s{color} | {color:red} root: The patch generated 6 new + 195 unchanged - 2
fixed = 201 total (was 197) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 39f1e82 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8989/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8989/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8989/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8989/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch,
> HIVE-18350.2.patch, HIVE-18350.3.patch, HIVE-18350.4.patch,
> HIVE-18350.5.patch, HIVE-18350.6.patch, HIVE-18350.7.patch,
> HIVE-18350.8.patch, HIVE-18350.9.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc.
> However, the load data uses the input file name. That results in inconsistent
> naming convention which makes SMB joins difficult in some scenarios and may
> cause trouble for other types of queries in future.
> We need consistent naming convention.
> For
[jira] [Commented] (HIVE-18541) Secure HS2 web UI with PAM
[ https://issues.apache.org/jira/browse/HIVE-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350531#comment-16350531 ] Oleksiy Sayankin commented on HIVE-18541: - [~lskuff], [~jxiang], [~szehon], guys please take a look at the patch. > Secure HS2 web UI with PAM > -- > > Key: HIVE-18541 > URL: https://issues.apache.org/jira/browse/HIVE-18541 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Oleksiy Sayankin >Assignee: Oleksiy Sayankin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18541.1.patch > > > Secure HS2 web UI with PAM. Add two new properties > * hive.server2.webui.use.pam > * Default value: false > * Description: If true, the HiveServer2 WebUI will be secured with PAM > * hive.server2.webui.pam.authenticator > * Default value: org.apache.hive.http.security.PamAuthenticator > * Description: Class for PAM authentication -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18565) Set some timeout to all CliDriver tests
[ https://issues.apache.org/jira/browse/HIVE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bertalan Kondrat reassigned HIVE-18565: --- Assignee: Bertalan Kondrat > Set some timeout to all CliDriver tests > --- > > Key: HIVE-18565 > URL: https://issues.apache.org/jira/browse/HIVE-18565 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Zoltan Haindrich >Assignee: Bertalan Kondrat >Priority: Major > > if a testcase runs into an infinite loop or something ; it should still fail > instead of causing timeout at the ptest executor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350521#comment-16350521 ] Zoltan Haindrich commented on HIVE-18578: - +1 > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18510) Enable running checkstyle on test sources as well
[ https://issues.apache.org/jira/browse/HIVE-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350509#comment-16350509 ] Zoltan Haindrich commented on HIVE-18510: - I'm not entirely sure that this will help making the codebase better...I feel that people are already kinda overwhelmed with these checkstyle things...people are rarely even write real unit testsso I feel that demanding checkstyle quality on tests will just probably take away peoples appetite to even start writing one... > Enable running checkstyle on test sources as well > - > > Key: HIVE-18510 > URL: https://issues.apache.org/jira/browse/HIVE-18510 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-18510.0.patch > > > Currently only source files are in the scope of checkstyle testing. We should > expand the scope to include our testing code as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18612) Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
[
https://issues.apache.org/jira/browse/HIVE-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-18612 started by Adam Szita.
-
> Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
> --
>
> Key: HIVE-18612
> URL: https://issues.apache.org/jira/browse/HIVE-18612
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
>
> As per this jira comment made by Yetus maven plugins that want to use
> {{java}} executable are seeing a 1.7 java binary. In this particular case
> Yetus sets JAVA_HOME to a 1.8 JDK installation, and thus maven uses that, but
> any subsequent {{java}} executes will use the JRE which they see on PATH.
> This should be fixed by adding the proper java/bin (that of JAVA_HOME
> setting) to PATH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18612) Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
[
https://issues.apache.org/jira/browse/HIVE-18612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita reassigned HIVE-18612:
-
> Build subprocesses under Yetus in Ptest use 1.7 jre instead of 1.8
> --
>
> Key: HIVE-18612
> URL: https://issues.apache.org/jira/browse/HIVE-18612
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
>
> As per this jira comment made by Yetus maven plugins that want to use
> {{java}} executable are seeing a 1.7 java binary. In this particular case
> Yetus sets JAVA_HOME to a 1.8 JDK installation, and thus maven uses that, but
> any subsequent {{java}} executes will use the JRE which they see on PATH.
> This should be fixed by adding the proper java/bin (that of JAVA_HOME
> setting) to PATH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18607) HBase HFile write does strange things
[
https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350395#comment-16350395
]
Hive QA commented on HIVE-18607:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908890/HIVE-18607.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12918 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_handler_bulk]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8988/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8988/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8988/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908890 - PreCommit-HIVE-Build
> HBase HFile write does strange things
> -
>
> Key: HIVE-18607
> URL: https://issues.apache.org/jira/browse/HIVE-18607
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18607.patch
>
>
> I cannot get the HBaseCliDriver to run locally, so first I'll use HiveQA to
> check smth.
> There's some strange code in the output handler that changes output directory
> into a file because Hive supposedly wants that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-18467: --- Attachment: (was: HIVE-18467.4.patch) > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch, > HIVE-18467.2.patch, HIVE-18467.5.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the REPL > DUMP commands, we should allow dumping of all databases using the "*" option > as in > _REPL DUMP *_ > On the repl load side there will not be an option to specify the database > name when loading from a location used to dump multiple databases, hence only > _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_ > Additionally, incremental dumps will go through all events across databases > in a warehouse and hence CREATE / DROP Database events have to be serialized > correctly to allow repl load to create them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
