[jira] [Commented] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302258#comment-16302258
]
Hive QA commented on HIVE-17495:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore: The patch generated 10 new +
1131 unchanged - 31 fixed = 1141 total (was 1162) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8372/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore itests/hcatalog-unit itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8372/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
> some code to fix that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[
https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302248#comment-16302248
]
Hive QA commented on HIVE-18335:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903477/HIVE-18335.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 11544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_create_no_grant]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastBytesHashMap.testOutOfBounds
(batchId=283)
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.TestVectorMapJoinFastLongHashMap.testOutOfBounds
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
(batchId=229)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8371/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8371/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8371/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903477 - PreCommit-HIVE-Build
> Vectorization : Check bounds of array before the allocation in
> VectorMapJoinFastBytesHashTable
> --
>
> Key: HIVE-18335
> URL: https://issues.apache.org/jira/browse/HIVE-18335
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch
>
>
> VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds
> before allocating array.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[
https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302232#comment-16302232
]
Hive QA commented on HIVE-18335:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8371/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization : Check bounds of array before the allocation in
> VectorMapJoinFastBytesHashTable
> --
>
> Key: HIVE-18335
> URL: https://issues.apache.org/jira/browse/HIVE-18335
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch
>
>
> VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds
> before allocating array.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18273) add LLAP-level counters for WM
[
https://issues.apache.org/jira/browse/HIVE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1630#comment-1630
]
Hive QA commented on HIVE-18273:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903472/HIVE-18273.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8370/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8370/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8370/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903472 - PreCommit-HIVE-Build
> add LLAP-level counters for WM
> --
>
> Key: HIVE-18273
> URL: https://issues.apache.org/jira/browse/HIVE-18273
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18273.01.patch, HIVE-18273.02.patch,
> HIVE-18273.patch
>
>
> On query fragment level (like IO counters)
> time queued as guaranteed;
> time running as guaranteed;
> time running as speculative.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18273) add LLAP-level counters for WM
[
https://issues.apache.org/jira/browse/HIVE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302208#comment-16302208
]
Hive QA commented on HIVE-18273:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} llap-common: The patch generated 2 new + 0 unchanged -
0 fixed = 2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
31s{color} | {color:red} ql: The patch generated 4 new + 67 unchanged - 0 fixed
= 71 total (was 67) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 11 new + 206
unchanged - 1 fixed = 217 total (was 207) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8370/yetus/diff-checkstyle-llap-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8370/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8370/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-common ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8370/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add LLAP-level counters for WM
> --
>
> Key: HIVE-18273
> URL: https://issues.apache.org/jira/browse/HIVE-18273
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18273.01.patch, HIVE-18273.02.patch,
> HIVE-18273.patch
>
>
> On query fragment level (like IO counters)
> time queued as guaranteed;
> time running as guaranteed;
> time running as speculative.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302193#comment-16302193
]
Hive QA commented on HIVE-17495:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903473/HIVE-17495.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 11542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartition
(batchId=213)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartition
(batchId=215)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition
(batchId=211)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartition
(batchId=210)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartition
(batchId=220)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testDatabaseOps
(batchId=201)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testPartitionOps
(batchId=201)
org.apache.hadoop.hive.metastore.cache.TestCachedStore.testTableOps
(batchId=201)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8369/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8369/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8369/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903473 - PreCommit-HIVE-Build
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be
[jira] [Commented] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302175#comment-16302175
]
Hive QA commented on HIVE-17495:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore: The patch generated 10 new +
1131 unchanged - 31 fixed = 1141 total (was 1162) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8369/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore itests/hcatalog-unit itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8369/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
> some code to fix that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them
[
https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302132#comment-16302132
]
Hive QA commented on HIVE-18326:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903470/HIVE-18326.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 11542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_alter_table_exchange_partition_fail]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8366/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8366/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8366/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903470 - PreCommit-HIVE-Build
> LLAP Tez scheduler - only preempt tasks if there's a dependency between them
>
>
> Key: HIVE-18326
> URL: https://issues.apache.org/jira/browse/HIVE-18326
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18326.01.patch, HIVE-18326.patch
>
>
> It is currently possible for e.g. two sides of a union (or a join for that
> matter) to have slightly different priorities. We don't want to preempt
> running tasks on one side in favor of the other side in such cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18269) LLAP: Fast llap io with slow processing pipeline can lead to OOM
[ https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18269: --- Assignee: Sergey Shelukhin (was: Prasanth Jayachandran) > LLAP: Fast llap io with slow processing pipeline can lead to OOM > > > Key: HIVE-18269 > URL: https://issues.apache.org/jira/browse/HIVE-18269 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Sergey Shelukhin > Attachments: HIVE-18269.1.patch, Screen Shot 2017-12-13 at 1.15.16 > AM.png > > > pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow > indefinitely when Llap IO is faster than processing pipeline. Since we don't > have backpressure to slow down the IO, this can lead to indefinite growth of > pending data leading to severe GC pressure and eventually lead to OOM. > This specific instance of LLAP was running on HDFS on top of EBS volume > backed by SSD. The query that triggered this is issue was ANALYZE STATISTICS > .. FOR COLUMNS which also gather bitvectors. Fast IO and Slow processing case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18335: -- Attachment: HIVE-18335.2.patch Added the similar logic to VectorMapJoinFastLongHashTable class as well. > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18335.1.patch, HIVE-18335.2.patch > > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18269) LLAP: Fast llap io with slow processing pipeline can lead to OOM
[ https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302114#comment-16302114 ] Sergey Shelukhin commented on HIVE-18269: - Update: we've seen large queue also producing GC problems without getting close to OOM with many decimal columns. The temp patch to see if the limit works performed well with queue size of 3-10, which I suspect will be insufficient for a cloud FS like S3 if IO thread is blocked - if pipeline can process 10 VRBs rapidly, it will have to wait for a while until the unblocked S3 reader produces more data and blocks, then process it quickly again and block, etc. This might require some testing. There are 3 possible approaches that I see: 1) Don't block physical reads from FS, but only block the decoding/etc. that produces java objects. That may be a complex threading change and/or would require separate throttle for the buffers (that may be more forgiving) lest they cause OOM. 2) Determine queue size dynamically based on speed of processing - e.g. start high, then see how fast next calls are coming and how fast IO is putting stuff in queue, and adjust down if IO is much faster; or start low (~10) and expand aggressively every time the next() waits (meaning IO is not keeping up). This is rather complex although may be the best long term solution. 3) Determine queue size per fragment (vertex, really) based on schema. Configure a high default limit (e.g. 10k to prevent OOMs), and the lower bound of the limit (e.g. 10). Then, at init time start with the limit as the high boundary, and reduce it based on the number and type of VRB vectors (reduce proportionally assuming the maximum limit is for a single INT vector, and it can never go below the minimum). This is hand wavy but easy to implement and reason about, and as a fail safe one can always set min=max to fix the queue size. I think we can start with 3 and consider 2 later. 1 is only good if we decide to separate FS and decoding threads that was a plan long time ago that was not implemented. [~gopalv] [~prasanth_j] [~hagleitn] any input? > LLAP: Fast llap io with slow processing pipeline can lead to OOM > > > Key: HIVE-18269 > URL: https://issues.apache.org/jira/browse/HIVE-18269 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18269.1.patch, Screen Shot 2017-12-13 at 1.15.16 > AM.png > > > pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow > indefinitely when Llap IO is faster than processing pipeline. Since we don't > have backpressure to slow down the IO, this can lead to indefinite growth of > pending data leading to severe GC pressure and eventually lead to OOM. > This specific instance of LLAP was running on HDFS on top of EBS volume > backed by SSD. The query that triggered this is issue was ANALYZE STATISTICS > .. FOR COLUMNS which also gather bitvectors. Fast IO and Slow processing case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18269) LLAP: Fast llap io with slow processing pipeline can lead to OOM
[ https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302114#comment-16302114 ] Sergey Shelukhin edited comment on HIVE-18269 at 12/22/17 11:54 PM: Update: we've seen large queue also producing GC problems without getting close to OOM with many decimal columns. The temp patch to see if the limit works performed well with queue size of 3-10, which I suspect will be insufficient for a cloud FS like S3 if IO thread is blocked - if pipeline can process 10 VRBs rapidly, it will have to wait for a while until the unblocked S3 reader produces more data and blocks, then process it quickly again and block, etc. This might require some testing. There are 3 possible approaches that I see: 1) Don't block physical reads from FS, but only block the decoding/etc. that produces java objects. That may be a complex threading change and/or would require separate throttle for the buffers (that may be more forgiving) lest they cause OOM. 2) Determine queue size dynamically based on speed of processing - e.g. start high, then see how fast next calls are coming and how fast IO is putting stuff in queue, and adjust down if IO is much faster; or start low (~10) and expand aggressively every time the next() waits (meaning IO is not keeping up). This is rather complex although may be the best long term solution. 3) Determine queue size per fragment (vertex, really) based on schema. Configure a high default limit (e.g. 10k to prevent OOMs), and the lower bound of the limit (e.g. 10). Then, at init time start with the limit as the high boundary, and reduce it based on the number and type of VRB vectors (reduce proportionally assuming the maximum limit is for a single INT vector, and it can never go below the minimum). So, one int column may store 10k vrbs but 50 decimal columns may store 10. This is hand wavy but easy to implement and reason about, and as a fail safe one can always set min=max to fix the queue size. I think we can start with 3 and consider 2 later. 1 is only good if we decide to separate FS and decoding threads that was a plan long time ago that was not implemented. [~gopalv] [~prasanth_j] [~hagleitn] any input? was (Author: sershe): Update: we've seen large queue also producing GC problems without getting close to OOM with many decimal columns. The temp patch to see if the limit works performed well with queue size of 3-10, which I suspect will be insufficient for a cloud FS like S3 if IO thread is blocked - if pipeline can process 10 VRBs rapidly, it will have to wait for a while until the unblocked S3 reader produces more data and blocks, then process it quickly again and block, etc. This might require some testing. There are 3 possible approaches that I see: 1) Don't block physical reads from FS, but only block the decoding/etc. that produces java objects. That may be a complex threading change and/or would require separate throttle for the buffers (that may be more forgiving) lest they cause OOM. 2) Determine queue size dynamically based on speed of processing - e.g. start high, then see how fast next calls are coming and how fast IO is putting stuff in queue, and adjust down if IO is much faster; or start low (~10) and expand aggressively every time the next() waits (meaning IO is not keeping up). This is rather complex although may be the best long term solution. 3) Determine queue size per fragment (vertex, really) based on schema. Configure a high default limit (e.g. 10k to prevent OOMs), and the lower bound of the limit (e.g. 10). Then, at init time start with the limit as the high boundary, and reduce it based on the number and type of VRB vectors (reduce proportionally assuming the maximum limit is for a single INT vector, and it can never go below the minimum). So, one int column may store 10k vrbs by 50 decimal columns may store 10. This is hand wavy but easy to implement and reason about, and as a fail safe one can always set min=max to fix the queue size. I think we can start with 3 and consider 2 later. 1 is only good if we decide to separate FS and decoding threads that was a plan long time ago that was not implemented. [~gopalv] [~prasanth_j] [~hagleitn] any input? > LLAP: Fast llap io with slow processing pipeline can lead to OOM > > > Key: HIVE-18269 > URL: https://issues.apache.org/jira/browse/HIVE-18269 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18269.1.patch, Screen Shot 2017-12-13 at 1.15.16 > AM.png > > > pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow > indefinitely when Llap IO is faster than processing
[jira] [Comment Edited] (HIVE-18269) LLAP: Fast llap io with slow processing pipeline can lead to OOM
[ https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302114#comment-16302114 ] Sergey Shelukhin edited comment on HIVE-18269 at 12/22/17 11:54 PM: Update: we've seen large queue also producing GC problems without getting close to OOM with many decimal columns. The temp patch to see if the limit works performed well with queue size of 3-10, which I suspect will be insufficient for a cloud FS like S3 if IO thread is blocked - if pipeline can process 10 VRBs rapidly, it will have to wait for a while until the unblocked S3 reader produces more data and blocks, then process it quickly again and block, etc. This might require some testing. There are 3 possible approaches that I see: 1) Don't block physical reads from FS, but only block the decoding/etc. that produces java objects. That may be a complex threading change and/or would require separate throttle for the buffers (that may be more forgiving) lest they cause OOM. 2) Determine queue size dynamically based on speed of processing - e.g. start high, then see how fast next calls are coming and how fast IO is putting stuff in queue, and adjust down if IO is much faster; or start low (~10) and expand aggressively every time the next() waits (meaning IO is not keeping up). This is rather complex although may be the best long term solution. 3) Determine queue size per fragment (vertex, really) based on schema. Configure a high default limit (e.g. 10k to prevent OOMs), and the lower bound of the limit (e.g. 10). Then, at init time start with the limit as the high boundary, and reduce it based on the number and type of VRB vectors (reduce proportionally assuming the maximum limit is for a single INT vector, and it can never go below the minimum). So, one int column may store 10k vrbs by 50 decimal columns may store 10. This is hand wavy but easy to implement and reason about, and as a fail safe one can always set min=max to fix the queue size. I think we can start with 3 and consider 2 later. 1 is only good if we decide to separate FS and decoding threads that was a plan long time ago that was not implemented. [~gopalv] [~prasanth_j] [~hagleitn] any input? was (Author: sershe): Update: we've seen large queue also producing GC problems without getting close to OOM with many decimal columns. The temp patch to see if the limit works performed well with queue size of 3-10, which I suspect will be insufficient for a cloud FS like S3 if IO thread is blocked - if pipeline can process 10 VRBs rapidly, it will have to wait for a while until the unblocked S3 reader produces more data and blocks, then process it quickly again and block, etc. This might require some testing. There are 3 possible approaches that I see: 1) Don't block physical reads from FS, but only block the decoding/etc. that produces java objects. That may be a complex threading change and/or would require separate throttle for the buffers (that may be more forgiving) lest they cause OOM. 2) Determine queue size dynamically based on speed of processing - e.g. start high, then see how fast next calls are coming and how fast IO is putting stuff in queue, and adjust down if IO is much faster; or start low (~10) and expand aggressively every time the next() waits (meaning IO is not keeping up). This is rather complex although may be the best long term solution. 3) Determine queue size per fragment (vertex, really) based on schema. Configure a high default limit (e.g. 10k to prevent OOMs), and the lower bound of the limit (e.g. 10). Then, at init time start with the limit as the high boundary, and reduce it based on the number and type of VRB vectors (reduce proportionally assuming the maximum limit is for a single INT vector, and it can never go below the minimum). This is hand wavy but easy to implement and reason about, and as a fail safe one can always set min=max to fix the queue size. I think we can start with 3 and consider 2 later. 1 is only good if we decide to separate FS and decoding threads that was a plan long time ago that was not implemented. [~gopalv] [~prasanth_j] [~hagleitn] any input? > LLAP: Fast llap io with slow processing pipeline can lead to OOM > > > Key: HIVE-18269 > URL: https://issues.apache.org/jira/browse/HIVE-18269 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-18269.1.patch, Screen Shot 2017-12-13 at 1.15.16 > AM.png > > > pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow > indefinitely when Llap IO is faster than processing pipeline. Since we don't > have backpressure to slow down the IO, this can lead
[jira] [Commented] (HIVE-18294) add switch to make acid table the default
[ https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302102#comment-16302102 ] Alan Gates commented on HIVE-18294: --- +1, looks good. > add switch to make acid table the default > - > > Key: HIVE-18294 > URL: https://issues.apache.org/jira/browse/HIVE-18294 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch, > HIVE-18294.04.patch, HIVE-18294.05.patch > > > it would be convenient for testing to have a switch that enables the behavior > where all suitable table tables (currently ORC + not sorted) are > automatically created with transactional=true, ie. full acid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18335: -- Attachment: HIVE-18335.1.patch [~mmccline] can you please review? > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > Attachments: HIVE-18335.1.patch > > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18335: -- Status: Patch Available (was: Open) > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[ https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18331: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Slim! > Renew the Kerberos ticket used by Druid Query runner > > > Key: HIVE-18331 > URL: https://issues.apache.org/jira/browse/HIVE-18331 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra > Fix For: 3.0.0 > > Attachments: HIVE-18331.patch > > > Druid Http Client has to renew the current user Kerberos ticket when it is > close to expire. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18096) add a user-friendly show plan command
[ https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302094#comment-16302094 ] Sergey Shelukhin commented on HIVE-18096: - [~harishjp] do you have any update on this? > add a user-friendly show plan command > - > > Key: HIVE-18096 > URL: https://issues.apache.org/jira/browse/HIVE-18096 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > > For admin to be able to get an overview of a resource plan. > We need to try to do this using sysdb. > If that is not possible to do in a nice way, we'd do a text-based one like > query explain, or desc extended table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18335) Vectorization : Check bounds of array before the allocation in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-18335: - > Vectorization : Check bounds of array before the allocation in > VectorMapJoinFastBytesHashTable > -- > > Key: HIVE-18335 > URL: https://issues.apache.org/jira/browse/HIVE-18335 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal > > VectorMapJoinFastBytesHashTable.allocateBucketArray() needs to check bounds > before allocating array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them
[
https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302093#comment-16302093
]
Hive QA commented on HIVE-18326:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-tez: The patch generated 6 new + 146 unchanged -
0 fixed = 152 total (was 146) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 5cd047b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8366/yetus/diff-checkstyle-llap-tez.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8366/yetus/whitespace-eol.txt
|
| modules | C: common llap-tez U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8366/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP Tez scheduler - only preempt tasks if there's a dependency between them
>
>
> Key: HIVE-18326
> URL: https://issues.apache.org/jira/browse/HIVE-18326
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18326.01.patch, HIVE-18326.patch
>
>
> It is currently possible for e.g. two sides of a union (or a join for that
> matter) to have slightly different priorities. We don't want to preempt
> running tasks on one side in favor of the other side in such cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[
https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-17495:
Description:
Only when CachedStore is enabled, we would like to make the following
optimizations:
1. During CachedStore prewarm, use one sql call to retrieve column stats
objects for a db and store it in the cache.
2. Cache some aggregate stats (e.g. aggregate stats for all partitions, which
seems to be commonly used) for query compilation speedup.
3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use an
iterator.next w/o checking with iterator.hasNext. This patch refactors some
code to fix that.
was:
1. One sql call to retrieve column stats objects for a db
2. Cache some aggregate stats for speedup
> CachedStore: prewarm improvement (avoid multiple sql calls to read partition
> column stats), refactoring and caching some aggregate stats
>
>
> Key: HIVE-17495
> URL: https://issues.apache.org/jira/browse/HIVE-17495
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch,
> HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch,
> HIVE-17495.6.patch, HIVE-17495.7.patch
>
>
> Only when CachedStore is enabled, we would like to make the following
> optimizations:
> 1. During CachedStore prewarm, use one sql call to retrieve column stats
> objects for a db and store it in the cache.
> 2. Cache some aggregate stats (e.g. aggregate stats for all partitions,
> which seems to be commonly used) for query compilation speedup.
> 3. There was a bug in {{MetaStoreUtils.aggrPartitionStats}}, which would use
> an iterator.next w/o checking with iterator.hasNext. This patch refactors
> some code to fix that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17495) CachedStore: prewarm improvement (avoid multiple sql calls to read partition column stats), refactoring and caching some aggregate stats
[ https://issues.apache.org/jira/browse/HIVE-17495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-17495: Attachment: HIVE-17495.7.patch > CachedStore: prewarm improvement (avoid multiple sql calls to read partition > column stats), refactoring and caching some aggregate stats > > > Key: HIVE-17495 > URL: https://issues.apache.org/jira/browse/HIVE-17495 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-17495.1.patch, HIVE-17495.2.patch, > HIVE-17495.3.patch, HIVE-17495.4.patch, HIVE-17495.5.patch, > HIVE-17495.6.patch, HIVE-17495.7.patch > > > 1. One sql call to retrieve column stats objects for a db > 2. Cache some aggregate stats for speedup -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18273) add LLAP-level counters for WM
[ https://issues.apache.org/jira/browse/HIVE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18273: Attachment: HIVE-18273.02.patch Rebased the patch. [~prasanth_j] can you take a look? > add LLAP-level counters for WM > -- > > Key: HIVE-18273 > URL: https://issues.apache.org/jira/browse/HIVE-18273 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18273.01.patch, HIVE-18273.02.patch, > HIVE-18273.patch > > > On query fragment level (like IO counters) > time queued as guaranteed; > time running as guaranteed; > time running as speculative. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them
[ https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18326: Attachment: HIVE-18326.01.patch Updated the patch. With some added logging (that is not included in the final patch) I can see on a union query the preemption is prevented between two sides of the union. > LLAP Tez scheduler - only preempt tasks if there's a dependency between them > > > Key: HIVE-18326 > URL: https://issues.apache.org/jira/browse/HIVE-18326 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18326.01.patch, HIVE-18326.patch > > > It is currently possible for e.g. two sides of a union (or a join for that > matter) to have slightly different priorities. We don't want to preempt > running tasks on one side in favor of the other side in such cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18294) add switch to make acid table the default
[ https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302051#comment-16302051 ] Eugene Koifman commented on HIVE-18294: --- patch 5 removes the prop from HiveConf. > add switch to make acid table the default > - > > Key: HIVE-18294 > URL: https://issues.apache.org/jira/browse/HIVE-18294 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch, > HIVE-18294.04.patch, HIVE-18294.05.patch > > > it would be convenient for testing to have a switch that enables the behavior > where all suitable table tables (currently ORC + not sorted) are > automatically created with transactional=true, ie. full acid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18294) add switch to make acid table the default
[ https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18294: -- Attachment: HIVE-18294.05.patch > add switch to make acid table the default > - > > Key: HIVE-18294 > URL: https://issues.apache.org/jira/browse/HIVE-18294 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch, > HIVE-18294.04.patch, HIVE-18294.05.patch > > > it would be convenient for testing to have a switch that enables the behavior > where all suitable table tables (currently ORC + not sorted) are > automatically created with transactional=true, ie. full acid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them
[ https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301983#comment-16301983 ] Sergey Shelukhin commented on HIVE-18326: - Hmm, I checked on the cluster and that is actually correct... DAG info is null everywhere including start call. That is not great cause we'd need sync to init this elsewhere. > LLAP Tez scheduler - only preempt tasks if there's a dependency between them > > > Key: HIVE-18326 > URL: https://issues.apache.org/jira/browse/HIVE-18326 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18326.patch > > > It is currently possible for e.g. two sides of a union (or a join for that > matter) to have slightly different priorities. We don't want to preempt > running tasks on one side in favor of the other side in such cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18095) add a unmanaged flag to triggers (applies to container based sessions)
[ https://issues.apache.org/jira/browse/HIVE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18095: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the reviews! > add a unmanaged flag to triggers (applies to container based sessions) > -- > > Key: HIVE-18095 > URL: https://issues.apache.org/jira/browse/HIVE-18095 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18095.01.patch, HIVE-18095.02.patch, > HIVE-18095.nogen.patch, HIVE-18095.patch > > > cc [~prasanth_j] > It should be impossible to attach global triggers for pools. Setting global > flag should probably automatically remove attachments to pools. > Global triggers would only support actions that Tez supports (for simplicity; > also, for now, move doesn't make a lot of sense because the trigger would > apply again after the move). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18095) add a unmanaged flag to triggers (applies to container based sessions)
[ https://issues.apache.org/jira/browse/HIVE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301947#comment-16301947 ] Prasanth Jayachandran commented on HIVE-18095: -- +1 > add a unmanaged flag to triggers (applies to container based sessions) > -- > > Key: HIVE-18095 > URL: https://issues.apache.org/jira/browse/HIVE-18095 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18095.01.patch, HIVE-18095.02.patch, > HIVE-18095.nogen.patch, HIVE-18095.patch > > > cc [~prasanth_j] > It should be impossible to attach global triggers for pools. Setting global > flag should probably automatically remove attachments to pools. > Global triggers would only support actions that Tez supports (for simplicity; > also, for now, move doesn't make a lot of sense because the trigger would > apply again after the move). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18221) test acid default
[
https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301938#comment-16301938
]
Hive QA commented on HIVE-18221:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903449/HIVE-18221.18.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 718 failed/errored test(s), 9369 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[jira] [Commented] (HIVE-18294) add switch to make acid table the default
[ https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301935#comment-16301935 ] Alan Gates commented on HIVE-18294: --- You're correct that the value will have to be put in the conf file on both HS2 (or wherever the server code is running) and on the metastore (assuming they are running on separate servers). But if the correct value is in the config file the MetastoreConf methods will properly extract it. They were designed with exactly this in mind, so that we don't have to duplicate the conf enums. So you should be able to use MetastoreConf.getVar() to pull a value out hive-site.xml. > add switch to make acid table the default > - > > Key: HIVE-18294 > URL: https://issues.apache.org/jira/browse/HIVE-18294 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch, > HIVE-18294.04.patch > > > it would be convenient for testing to have a switch that enables the behavior > where all suitable table tables (currently ORC + not sorted) are > automatically created with transactional=true, ie. full acid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18095) add a unmanaged flag to triggers (applies to container based sessions)
[ https://issues.apache.org/jira/browse/HIVE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301933#comment-16301933 ] Sergey Shelukhin commented on HIVE-18095: - Failures are not related. The only new one is llap_smb. Looks like sysdb has extra changes that are caused by some previous patches, in non WM related areas; I fixed the WM diff. [~prasanth_j] [~harishjp] reviewed above; can you also +1? :) > add a unmanaged flag to triggers (applies to container based sessions) > -- > > Key: HIVE-18095 > URL: https://issues.apache.org/jira/browse/HIVE-18095 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18095.01.patch, HIVE-18095.02.patch, > HIVE-18095.nogen.patch, HIVE-18095.patch > > > cc [~prasanth_j] > It should be impossible to attach global triggers for pools. Setting global > flag should probably automatically remove attachments to pools. > Global triggers would only support actions that Tez supports (for simplicity; > also, for now, move doesn't make a lot of sense because the trigger would > apply again after the move). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18257) implement scheduling policy configuration instead of hardcoding fair scheduling
[ https://issues.apache.org/jira/browse/HIVE-18257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18257: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master after a trivial resourceplan file update (new commands were added in other patches that had old scheduling policy names in output). > implement scheduling policy configuration instead of hardcoding fair > scheduling > --- > > Key: HIVE-18257 > URL: https://issues.apache.org/jira/browse/HIVE-18257 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18257.01.patch, HIVE-18257.02.patch, > HIVE-18257.02.patch, HIVE-18257.03.patch, HIVE-18257.patch > > > Not sure it makes sense to actually make it pluggable. At least the standard > ones will be an enum; we don't expect people to implement custom classes - > phase 2 if someone wants to -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18221) test acid default
[
https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301895#comment-16301895
]
Hive QA commented on HIVE-18221:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} common: The patch generated 1 new + 931 unchanged - 0
fixed = 932 total (was 931) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} standalone-metastore: The patch generated 2 new + 210
unchanged - 0 fixed = 212 total (was 210) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 1094 unchanged - 0
fixed = 1096 total (was 1094) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / e2e1139 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8365/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8365/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8365/yetus/diff-checkstyle-ql.txt
|
| modules | C: common standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8365/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> test acid default
> -
>
> Key: HIVE-18221
> URL: https://issues.apache.org/jira/browse/HIVE-18221
> Project: Hive
> Issue Type: Test
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch,
> HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch,
> HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch,
> HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch,
> HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18334) Cannot JOIN ON result of COALESCE
[
https://issues.apache.org/jira/browse/HIVE-18334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Jackson updated HIVE-18334:
Description:
A join is returning no results when the ON clause is equating the results of
two COALESCE functions. To reproduce:
{code:SQL}
CREATE TABLE t5 (
dno INTEGER,
dname VARCHAR(30),
eno INTEGER,
ename VARCHAR(30));
CREATE TABLE t6 (
dno INTEGER,
dname VARCHAR(30),
eno INTEGER,
ename VARCHAR(30));
INSERT INTO t5 VALUES
(10, 'FOO', NULL, NULL),
(20, 'BAR', NULL, NULL),
(NULL, NULL, 7300, 'LARRY'),
(NULL, NULL, 7400, 'MOE'),
(NULL, NULL, 7500, 'CURLY');
INSERT INTO t6 VALUES
(10, 'LENNON', NULL, NULL),
(20, 'MCCARTNEY', NULL, NULL),
(NULL, NULL, 7300, 'READY'),
(NULL, NULL, 7400, 'WILLING'),
(NULL, NULL, 7500, 'ABLE');
-- Fails with 0 results
SELECT *
FROM t5
INNER JOIN t6
ON COALESCE(`t5`.`eno`, `t5`.`dno`) = COALESCE(`t6`.`eno`, `t6`.`dno`)
-- Full cross with where clause works (in nonstrict mode), returning 5 results
SELECT *
FROM t5
JOIN t6
WHERE `t5`.`eno` = `t6`.`eno` OR `t5`.`dno` = `t6`.`dno`
-- Strange that coalescing the same field returns 2 results...
SELECT *
FROM t5
INNER JOIN t6
ON COALESCE(`t5`.`dno`, `t5`.`dno`) = COALESCE(`t6`.`dno`, `t6`.`dno`)
-- ...and coalescing the other field returns 3 results
SELECT *
FROM t5
INNER JOIN t6
ON COALESCE(`t5`.`eno`, `t5`.`eno`) = COALESCE(`t6`.`eno`, `t6`.`eno`)
{code}
was:
A join is returning no results when the ON clause is equating the results of
two COALESCE functions. To reproduce:
{code:SQL}
CREATE TABLE t5 (
dno INTEGER,
dname VARCHAR(30),
eno INTEGER,
ename VARCHAR(30));
CREATE TABLE t6 (
dno INTEGER,
dname VARCHAR(30),
eno INTEGER,
ename VARCHAR(30));
INSERT INTO t5 VALUES
(10, 'FOO', NULL, NULL),
(20, 'BAR', NULL, NULL),
(NULL, NULL, 7300, 'LARRY'),
(NULL, NULL, 7400, 'MOE'),
(NULL, NULL, 7500, 'CURLY');
INSERT INTO t6 VALUES
(10, 'LENNON', NULL, NULL),
(20, 'MCCARTNEY', NULL, NULL),
(NULL, NULL, 7300, 'READY'),
(NULL, NULL, 7400, 'WILLING'),
(NULL, NULL, 7500, 'ABLE');
-- Fails with 0 results
SELECT *
FROM t5
INNER JOIN t6
ON COALESCE(`t5`.`eno`, `t5`.`dno`) = COALESCE(`t6`.`eno`, `t6`.`dno`)
-- Full cross with where clause works, returning 5 results
SELECT *
FROM t5
JOIN t6
WHERE `t5`.`eno` = `t6`.`eno` OR `t5`.`dno` = `t6`.`dno`
-- Strange that coalescing the same field returns 2 results...
SELECT *
FROM t5
INNER JOIN t6
ON COALESCE(`t5`.`dno`, `t5`.`dno`) = COALESCE(`t6`.`dno`, `t6`.`dno`)
-- ...and coalescing the other field returns 3 results
SELECT *
FROM t5
INNER JOIN t6
ON COALESCE(`t5`.`eno`, `t5`.`eno`) = COALESCE(`t6`.`eno`, `t6`.`eno`)
{code}
> Cannot JOIN ON result of COALESCE
> --
>
> Key: HIVE-18334
> URL: https://issues.apache.org/jira/browse/HIVE-18334
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
> Environment: Amazon AWS
> Release label:emr-5.11.0
> Hadoop distribution:Amazon 2.7.3
> Applications:Hive 2.3.2, Pig 0.17.0, Hue 4.0.1
> classification=hive-site,properties=[hive.strict.checks.cartesian.product=false,hive.mapred.mode=nonstrict]
>Reporter: Paul Jackson
>Priority: Minor
>
> A join is returning no results when the ON clause is equating the results of
> two COALESCE functions. To reproduce:
> {code:SQL}
> CREATE TABLE t5 (
> dno INTEGER,
> dname VARCHAR(30),
> eno INTEGER,
> ename VARCHAR(30));
> CREATE TABLE t6 (
> dno INTEGER,
> dname VARCHAR(30),
> eno INTEGER,
> ename VARCHAR(30));
> INSERT INTO t5 VALUES
> (10, 'FOO', NULL, NULL),
> (20, 'BAR', NULL, NULL),
> (NULL, NULL, 7300, 'LARRY'),
> (NULL, NULL, 7400, 'MOE'),
> (NULL, NULL, 7500, 'CURLY');
> INSERT INTO t6 VALUES
> (10, 'LENNON', NULL, NULL),
> (20, 'MCCARTNEY', NULL, NULL),
> (NULL, NULL, 7300, 'READY'),
> (NULL, NULL, 7400, 'WILLING'),
> (NULL, NULL, 7500, 'ABLE');
> -- Fails with 0 results
> SELECT *
> FROM t5
> INNER JOIN t6
> ON COALESCE(`t5`.`eno`, `t5`.`dno`) = COALESCE(`t6`.`eno`, `t6`.`dno`)
> -- Full cross with where clause works (in nonstrict mode), returning 5 results
> SELECT *
> FROM t5
> JOIN t6
> WHERE `t5`.`eno` = `t6`.`eno` OR `t5`.`dno` = `t6`.`dno`
> -- Strange that coalescing the same field returns 2 results...
> SELECT *
> FROM t5
> INNER JOIN t6
> ON COALESCE(`t5`.`dno`, `t5`.`dno`) = COALESCE(`t6`.`dno`, `t6`.`dno`)
> -- ...and coalescing the other field returns 3 results
> SELECT *
> FROM t5
> INNER JOIN t6
> ON COALESCE(`t5`.`eno`, `t5`.`eno`) = COALESCE(`t6`.`eno`, `t6`.`eno`)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18320) Support ACID Tables Replication
[ https://issues.apache.org/jira/browse/HIVE-18320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301883#comment-16301883 ] Sergey Shelukhin commented on HIVE-18320: - Will this also work for MM tables? We spent all this time converting them from writeId to txnId and now we'll convert them back I guess ;) cc [~ekoifman] [~steveyeom2017] > Support ACID Tables Replication > --- > > Key: HIVE-18320 > URL: https://issues.apache.org/jira/browse/HIVE-18320 > Project: Hive > Issue Type: New Feature > Components: HiveServer2, Metastore, repl, Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: anishek > Labels: ACID, DR, replication > Attachments: Hive Replication- ACID Tables.pdf > > > Currently, Full ACID and MM (Micro-Managed) tables are not supported for > replication. > Need to support it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18291) An exception should be raised if the result is outside the range of decimal
[
https://issues.apache.org/jira/browse/HIVE-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301874#comment-16301874
]
Sergey Shelukhin commented on HIVE-18291:
-
There's a JIRA for this somewhere already, and IIRC I even had a partial patch.
The problem as mentioned above is the default behavior. In the perfect world, a
config flag needs to be added, but given a number of places where that would
need to be checked, it's a big pain so seems noone has picked it up.
> An exception should be raised if the result is outside the range of decimal
> ---
>
> Key: HIVE-18291
> URL: https://issues.apache.org/jira/browse/HIVE-18291
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Marco Gaido
>Assignee: Daniel Voros
>
> Citing SQL:2011 on page 27 available at
> http://standards.iso.org/ittf/PubliclyAvailableStandards/c053681_ISO_IEC_9075-1_2011.zip:
> {noformat}
> If the result cannot be represented exactly in the result type, then whether
> it is rounded
> or truncated is implementation-defined. An exception condition is raised if
> the result is
> outside the range of numeric values of the result type, or if the arithmetic
> operation
> is not defined for the operands.
> {noformat}
> Currently Hive is returning NULL instead of throwing an exception if the
> result is out of range, eg.:
> {code}
> > select 100.01*100.01;
> +---+
> | _c0 |
> +---+
> | NULL |
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18293:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
committed to master
thanks Johannes for the contribution
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Johannes Alberti
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
>
[jira] [Commented] (HIVE-18294) add switch to make acid table the default
[ https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301868#comment-16301868 ] Eugene Koifman commented on HIVE-18294: --- I would prefer to only have the prop in MetastoreConf and the logic in TransactionalValidationListener - that would have been clean but there is some pushback on this. See HIVE-18285. Unfortunately, CTAS command writes the data first (this part needs to know if it's doing an Acid write) and then creates the metastore object which causes the Listener to run. So I was forced to also put the same logic in SemanticAnalyzer. I don't think the Conf object in SemanticAnalzyer has the MetastoreConf keys - but I would have run a test to make sure; it's a little hazy at this point. > add switch to make acid table the default > - > > Key: HIVE-18294 > URL: https://issues.apache.org/jira/browse/HIVE-18294 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch, > HIVE-18294.04.patch > > > it would be convenient for testing to have a switch that enables the behavior > where all suitable table tables (currently ORC + not sorted) are > automatically created with transactional=true, ie. full acid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[
https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301865#comment-16301865
]
Hive QA commented on HIVE-18331:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903432/HIVE-18331.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 11541 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenamePartitionWithCM
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8364/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8364/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8364/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903432 - PreCommit-HIVE-Build
> Renew the Kerberos ticket used by Druid Query runner
>
>
> Key: HIVE-18331
> URL: https://issues.apache.org/jira/browse/HIVE-18331
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
> Attachments: HIVE-18331.patch
>
>
> Druid Http Client has to renew the current user Kerberos ticket when it is
> close to expire.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18294) add switch to make acid table the default
[ https://issues.apache.org/jira/browse/HIVE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301850#comment-16301850 ] Alan Gates commented on HIVE-18294: --- Why mirror CREATE_TABLES_AS_ACID in HiveConf in addition to MetastoreConf? Can't everywhere use the Metastore value? That way we don't have to worry about different parts of code reading different values. I know I haven't cleaned up things in HiveConf yet, but I plan to deprecate all the HiveConf values mirrored in MetastoreConf and shift the Hive code to use the MetastoreConf values. Other than that, looks good. > add switch to make acid table the default > - > > Key: HIVE-18294 > URL: https://issues.apache.org/jira/browse/HIVE-18294 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18294.01.patch, HIVE-18294.03.patch, > HIVE-18294.04.patch > > > it would be convenient for testing to have a switch that enables the behavior > where all suitable table tables (currently ORC + not sorted) are > automatically created with transactional=true, ie. full acid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18329) There are 2 configs to detect/warn for cross products
[
https://issues.apache.org/jira/browse/HIVE-18329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301851#comment-16301851
]
Sergey Shelukhin commented on HIVE-18329:
-
The 2nd one used to be part of hive.mapred.mode that I've split into parts...
not sure where the 1st one comes from
> There are 2 configs to detect/warn for cross products
> -
>
> Key: HIVE-18329
> URL: https://issues.apache.org/jira/browse/HIVE-18329
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>
> The following 2 options seem to be after a very similar thing:
> {code}
> hive.exec.check.crossproducts
> hive.strict.checks.cartesian.product
> {code}
> not sure about the differences...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[ https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301847#comment-16301847 ] Sergey Shelukhin commented on HIVE-18331: - -0 on the design (discussed on IM about the scope of the keytab login etc.), +1 on the patch... > Renew the Kerberos ticket used by Druid Query runner > > > Key: HIVE-18331 > URL: https://issues.apache.org/jira/browse/HIVE-18331 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra > Attachments: HIVE-18331.patch > > > Druid Http Client has to renew the current user Kerberos ticket when it is > close to expire. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Attachment: HIVE-18221.18.patch > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, > HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, > HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, > HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Status: Patch Available (was: Open) > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, > HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, > HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, > HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18293) Hive is failing to compact tables contained within a folder that is not owned by identity running HiveMetaStore
[
https://issues.apache.org/jira/browse/HIVE-18293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301797#comment-16301797
]
Eugene Koifman commented on HIVE-18293:
---
failures are not related
+1
> Hive is failing to compact tables contained within a folder that is not owned
> by identity running HiveMetaStore
> ---
>
> Key: HIVE-18293
> URL: https://issues.apache.org/jira/browse/HIVE-18293
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.1.1
> Environment: Centos6.5/Hadoop2.7.4/Java7
>Reporter: Johannes Alberti
>Assignee: Johannes Alberti
>Priority: Critical
> Attachments: HIVE-18293.01.patch, HIVE-18293.patch
>
>
> ACID tables are not getting compacted properly due to an
> AccessControlException, this only occurs for tables contained in the
> non-default database.
> The root cause for the issue is the re-use of an already created
> DistributedFileSystem instance within a new DoAs context. I will attach a
> patch for the same.
> Stack below (anonymized)
> {noformat}
> compactor.Worker: Caught an exception in the main loop of compactor worker
> [[hostname]]-34, org.apache.hadoop.security.AccessControlException:
> Permission denied: user=hive, access=EXECUTE,
> inode="/hive/non-default.db":nothive:othergroup:drwxrwx---
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:215)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:199)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1752)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:100)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3820)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:855)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread$1.run(CompactorThread.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at
> org.apache.hadoop.hive.ql.txn.compactor.CompactorThread.findUserToRunAs(CompactorThread.java:169)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:151)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=hive,
[jira] [Commented] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[
https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301784#comment-16301784
]
Hive QA commented on HIVE-18331:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} druid-handler: The patch generated 0 new + 24
unchanged - 1 fixed = 24 total (was 25) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3aaddb2 |
| Default Java | 1.8.0_111 |
| modules | C: druid-handler U: druid-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8364/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Renew the Kerberos ticket used by Druid Query runner
>
>
> Key: HIVE-18331
> URL: https://issues.apache.org/jira/browse/HIVE-18331
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
> Attachments: HIVE-18331.patch
>
>
> Druid Http Client has to renew the current user Kerberos ticket when it is
> close to expire.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-13567) Enable auto-gather column stats by default
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301766#comment-16301766 ] Eugene Koifman commented on HIVE-13567: --- [~kgyrtkirk], I noticed that this patch explicitly disabled this option in TestAcidOnTez. Why is that? > Enable auto-gather column stats by default > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Zoltan Haindrich > Fix For: 3.0.0 > > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch, HIVE-13567.05.patch, > HIVE-13567.06.patch, HIVE-13567.07.patch, HIVE-13567.08.patch, > HIVE-13567.09.patch, HIVE-13567.10.patch, HIVE-13567.11.patch, > HIVE-13567.12.patch, HIVE-13567.13.patch, HIVE-13567.14.patch, > HIVE-13567.15.patch, HIVE-13567.16.patch, HIVE-13567.17.patch, > HIVE-13567.18.patch, HIVE-13567.19.patch, HIVE-13567.20.patch, > HIVE-13567.21.patch, HIVE-13567.22.patch, HIVE-13567.23.patch, > HIVE-13567.23wip01.patch, HIVE-13567.23wip02.patch, HIVE-13567.23wip03.patch, > HIVE-13567.23wip04.patch, HIVE-13567.23wip05.patch, HIVE-13567.23wip06.patch, > HIVE-13567.23wip07.patch, HIVE-13567.23wip08.patch, HIVE-13567.23wip09.patch, > HIVE-13567.23wip10.patch, HIVE-13567.24.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[ https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301765#comment-16301765 ] Ashutosh Chauhan commented on HIVE-18330: - AFAIK there are no users of this functionality. We may just @Ignore this test. > Fix TestMsgBusConnection - doesn't test tests the original intention > > > Key: HIVE-18330 > URL: https://issues.apache.org/jira/browse/HIVE-18330 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18330.01.patch > > > If the msgbus usage is configured; and the ActiveMQ broker is down; the > notificationlistener throws NPEs. > this test should have never been passed...there is a point where it drops a > database; and that command returns with an error - there are other things > which are intrestinglike create database on an existing db is sucess > somewhere - so it get posted to the msgbus. > discovered during HIVE-18238 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18108) in case basic stats are missing; rowcount estimation depends on the selected columns size
[
https://issues.apache.org/jira/browse/HIVE-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301757#comment-16301757
]
Hive QA commented on HIVE-18108:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903430/HIVE-18108.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11541 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_serde] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8363/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8363/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8363/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903430 - PreCommit-HIVE-Build
> in case basic stats are missing; rowcount estimation depends on the selected
> columns size
> -
>
> Key: HIVE-18108
> URL: https://issues.apache.org/jira/browse/HIVE-18108
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18108.01.patch, HIVE-18108.02.patch,
> HIVE-18108.02wip02.patch, HIVE-18108.03.patch
>
>
> in case basicstats are not available (especially rowcount):
> {code}
> set hive.stats.autogather=false;
> create table t (a integer, b string);
> insert into t values (1,'asd1');
> insert into t values (2,'asd2');
> insert into t values (3,'asd3');
> insert into t values (4,'asd4');
> insert into t values (5,'asd5');
> explain select a,count(1) from t group by a;
> -- estimated to read 8 rows from table t
> explain select b,count(1) from t group by b;
> -- estimated: 1 rows
> explain select a,b,count(1) from t group by a,b;
> -- estimated: 1 rows
> {code}
> it may not depend on the actually selected column set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301713#comment-16301713
]
liyunzhang commented on HIVE-18301:
---
[~xuefuz],[~csun]: I read jiras about MapInput IOContext problem and enable
MapInput rdd cache. And found the problem only happens OContext problem with
multiple MapWorks cloned for multi-insert \[Spark Branch\] like HIVE-8920
mentioned.
In HIVE-8920, I found the failure case is like
{code}
from (select * from dec union all select * from dec2) s
insert overwrite table dec3 select s.name, sum(s.value) group by s.name
insert overwrite table dec4 select s.name, s.value order by s.value;
{code}
I indeed saw the exception in my hive.log like
{code}
Caused by: java.lang.IllegalStateException: Invalid input path
hdfs://localhost:8020/user/hive/warehouse/dec2/dec.txt
at
org.apache.hadoop.hive.ql.exec.MapOperator.getNominalPath(MapOperator.java:406)
at
org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:442)
{code}
here the problem happens on the MapInput is the union result of dec and dec2
case. But when I modify case
{code}
from (select * from dec ) s
insert overwrite table dec3 select s.name, sum(s.value) group by s.name
insert overwrite table dec4 select s.name, s.value order by s.value;
{code}
No such exception whether in local or yarn mode.
Whether the problem only happens in such complicated case( the rdd cache is
the union result of two tables)? If only happen in such complicated case, why
not only disable MapInput rdd cache in such case? Is there any other reason to
disable MapInput#rdd cache? Please spend some time to view it as both of you
have experience on it, thanks!
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
> ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
> is null when rdd cach is enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18108) in case basic stats are missing; rowcount estimation depends on the selected columns size
[
https://issues.apache.org/jira/browse/HIVE-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301703#comment-16301703
]
Hive QA commented on HIVE-18108:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 1 new + 96 unchanged - 0 fixed
= 97 total (was 96) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 3aaddb2 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8363/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8363/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> in case basic stats are missing; rowcount estimation depends on the selected
> columns size
> -
>
> Key: HIVE-18108
> URL: https://issues.apache.org/jira/browse/HIVE-18108
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18108.01.patch, HIVE-18108.02.patch,
> HIVE-18108.02wip02.patch, HIVE-18108.03.patch
>
>
> in case basicstats are not available (especially rowcount):
> {code}
> set hive.stats.autogather=false;
> create table t (a integer, b string);
> insert into t values (1,'asd1');
> insert into t values (2,'asd2');
> insert into t values (3,'asd3');
> insert into t values (4,'asd4');
> insert into t values (5,'asd5');
> explain select a,count(1) from t group by a;
> -- estimated to read 8 rows from table t
> explain select b,count(1) from t group by b;
> -- estimated: 1 rows
> explain select a,b,count(1) from t group by a,b;
> -- estimated: 1 rows
> {code}
> it may not depend on the actually selected column set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301668#comment-16301668
]
Hive QA commented on HIVE-18238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903426/HIVE-18238.01wip01.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 11146 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[jira] [Updated] (HIVE-18053) Support different table types for MVs
[ https://issues.apache.org/jira/browse/HIVE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18053: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks for reviewing [~ashutoshc] > Support different table types for MVs > - > > Key: HIVE-18053 > URL: https://issues.apache.org/jira/browse/HIVE-18053 > Project: Hive > Issue Type: Sub-task > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18053.01.patch, HIVE-18053.patch > > > MVs backed by MM tables, managed tables, external tables. This might work > already, but we need to add tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[ https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-18331: - Assignee: slim bouguerra > Renew the Kerberos ticket used by Druid Query runner > > > Key: HIVE-18331 > URL: https://issues.apache.org/jira/browse/HIVE-18331 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra > Attachments: HIVE-18331.patch > > > Druid Http Client has to renew the current user Kerberos ticket when it is > close to expire. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[ https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18331: -- Attachment: HIVE-18331.patch > Renew the Kerberos ticket used by Druid Query runner > > > Key: HIVE-18331 > URL: https://issues.apache.org/jira/browse/HIVE-18331 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra > Attachments: HIVE-18331.patch > > > Druid Http Client has to renew the current user Kerberos ticket when it is > close to expire. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18331) Renew the Kerberos ticket used by Druid Query runner
[ https://issues.apache.org/jira/browse/HIVE-18331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18331: -- Status: Patch Available (was: Open) > Renew the Kerberos ticket used by Druid Query runner > > > Key: HIVE-18331 > URL: https://issues.apache.org/jira/browse/HIVE-18331 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra > Attachments: HIVE-18331.patch > > > Druid Http Client has to renew the current user Kerberos ticket when it is > close to expire. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18108) in case basic stats are missing; rowcount estimation depends on the selected columns size
[
https://issues.apache.org/jira/browse/HIVE-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18108:
Attachment: HIVE-18108.03.patch
> in case basic stats are missing; rowcount estimation depends on the selected
> columns size
> -
>
> Key: HIVE-18108
> URL: https://issues.apache.org/jira/browse/HIVE-18108
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18108.01.patch, HIVE-18108.02.patch,
> HIVE-18108.02wip02.patch, HIVE-18108.03.patch
>
>
> in case basicstats are not available (especially rowcount):
> {code}
> set hive.stats.autogather=false;
> create table t (a integer, b string);
> insert into t values (1,'asd1');
> insert into t values (2,'asd2');
> insert into t values (3,'asd3');
> insert into t values (4,'asd4');
> insert into t values (5,'asd5');
> explain select a,count(1) from t group by a;
> -- estimated to read 8 rows from table t
> explain select b,count(1) from t group by b;
> -- estimated: 1 rows
> explain select a,b,count(1) from t group by a,b;
> -- estimated: 1 rows
> {code}
> it may not depend on the actually selected column set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301610#comment-16301610
]
Hive QA commented on HIVE-18238:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 1 new + 189 unchanged - 0
fixed = 190 total (was 189) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hcatalog/core: The patch generated 0 new + 33
unchanged - 1 fixed = 33 total (was 34) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hcatalog-pig-adapter passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch server-extensions passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0e2dab9 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8362/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql hcatalog/core hcatalog/hcatalog-pig-adapter
hcatalog/server-extensions U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8362/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
>
[jira] [Updated] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18238:
Status: Patch Available (was: Open)
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18238:
Attachment: HIVE-18238.01wip01.patch
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18238.01wip01.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistics;
> select "${hiveconf:hive.mapred.mode}";
> {code}
> currently; the last select returns {{nonstrict}} because of
> [this|https://github.com/apache/hive/blob/7ddd915bf82a68c8ab73b0c4ca409f1a6d43d227/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1696]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[
https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301560#comment-16301560
]
Hive QA commented on HIVE-18330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903410/HIVE-18330.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11540 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8361/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8361/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8361/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903410 - PreCommit-HIVE-Build
> Fix TestMsgBusConnection - doesn't test tests the original intention
>
>
> Key: HIVE-18330
> URL: https://issues.apache.org/jira/browse/HIVE-18330
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18330.01.patch
>
>
> If the msgbus usage is configured; and the ActiveMQ broker is down; the
> notificationlistener throws NPEs.
> this test should have never been passed...there is a point where it drops a
> database; and that command returns with an error - there are other things
> which are intrestinglike create database on an existing db is sucess
> somewhere - so it get posted to the msgbus.
> discovered during HIVE-18238
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18226) handle UDF to double/int over aggregate
[
https://issues.apache.org/jira/browse/HIVE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-18226:
-
Assignee: slim bouguerra
> handle UDF to double/int over aggregate
> ---
>
> Key: HIVE-18226
> URL: https://issues.apache.org/jira/browse/HIVE-18226
> Project: Hive
> Issue Type: Sub-task
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>
> In cases like the following query Hive planner adds extra UDFtoDouble over
> integer columns.
> This kind of udf can be pushed to Druid as DoubleSum instead of LongSum and
> vice versa.
> {code}
> PREHOOK: query: EXPLAIN SELECT floor_year(`__time`), SUM(ctinyint)/ count(*)
> FROM druid_table GROUP BY floor_year(`__time`)
> PREHOOK: type: QUERY
> POSTHOOK: query: EXPLAIN SELECT floor_year(`__time`), SUM(ctinyint)/ count(*)
> FROM druid_table GROUP BY floor_year(`__time`)
> POSTHOOK: type: QUERY
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: druid_table
> properties:
> druid.query.json
> {"queryType":"timeseries","dataSource":"default.druid_table","descending":false,"granularity":"year","aggregations":[{"type":"longSum","name":"$f1","fieldName":"ctinyint"},{"type":"count","name":"$f2"}],"intervals":["1900-01-01T00:00:00.000/3000-01-01T00:00:00.000"],"context":{"skipEmptyBuckets":true}}
> druid.query.type timeseries
> Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> Select Operator
> expressions: __time (type: timestamp with local time zone),
> (UDFToDouble($f1) / UDFToDouble($f2)) (type: double)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 9173 Data size: 0 Basic stats: PARTIAL
> Column stats: NONE
> table:
> input format:
> org.apache.hadoop.mapred.SequenceFileInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[
https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301503#comment-16301503
]
Hive QA commented on HIVE-18330:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0e2dab9 |
| Default Java | 1.8.0_111 |
| modules | C: hcatalog/server-extensions U: hcatalog/server-extensions |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8361/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix TestMsgBusConnection - doesn't test tests the original intention
>
>
> Key: HIVE-18330
> URL: https://issues.apache.org/jira/browse/HIVE-18330
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18330.01.patch
>
>
> If the msgbus usage is configured; and the ActiveMQ broker is down; the
> notificationlistener throws NPEs.
> this test should have never been passed...there is a point where it drops a
> database; and that command returns with an error - there are other things
> which are intrestinglike create database on an existing db is sucess
> somewhere - so it get posted to the msgbus.
> discovered during HIVE-18238
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[ https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18330: Description: If the msgbus usage is configured; and the ActiveMQ broker is down; the notificationlistener throws NPEs. this test should have never been passed...there is a point where it drops a database; and that command returns with an error - there are other things which are intrestinglike create database on an existing db is sucess somewhere - so it get posted to the msgbus. discovered during HIVE-18238 was: If the msgbus usage is configured; and the ActiveMQ broker is down; the notificationlistener throws NPEs. this test should have never been passed...there is a point where it drops a database; and that command returns with an error - there are other things which are intrestinglike create database on an existing db is sucess somewhere - so it get posted to the msgbus. > Fix TestMsgBusConnection - doesn't test tests the original intention > > > Key: HIVE-18330 > URL: https://issues.apache.org/jira/browse/HIVE-18330 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18330.01.patch > > > If the msgbus usage is configured; and the ActiveMQ broker is down; the > notificationlistener throws NPEs. > this test should have never been passed...there is a point where it drops a > database; and that command returns with an error - there are other things > which are intrestinglike create database on an existing db is sucess > somewhere - so it get posted to the msgbus. > discovered during HIVE-18238 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[
https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301468#comment-16301468
]
Zoltan Haindrich commented on HIVE-18330:
-
I'm not sure which would be the preffered way to fix this:
# make the current behavior sacred: if the AMQ service is offline; no
metastore operations can be done => adjust test
# tolerate that the AMQ is offline in {{NotificationListener}} => adjust it
to not throw NPE
# are there any known users of this feature? It would be great to know that
they prefer the current logic or not...
[~thejas], [~ashutoshc]: I'm not sure which option is the preffered; I will go
with the 2. for now ; because all exceptions were just logged everywhere...so I
guess the original intention was to keep working in case AMQ is down.
> Fix TestMsgBusConnection - doesn't test tests the original intention
>
>
> Key: HIVE-18330
> URL: https://issues.apache.org/jira/browse/HIVE-18330
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18330.01.patch
>
>
> If the msgbus usage is configured; and the ActiveMQ broker is down; the
> notificationlistener throws NPEs.
> this test should have never been passed...there is a point where it drops a
> database; and that command returns with an error - there are other things
> which are intrestinglike create database on an existing db is sucess
> somewhere - so it get posted to the msgbus.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[ https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18330: Status: Patch Available (was: Open) > Fix TestMsgBusConnection - doesn't test tests the original intention > > > Key: HIVE-18330 > URL: https://issues.apache.org/jira/browse/HIVE-18330 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18330.01.patch > > > If the msgbus usage is configured; and the ActiveMQ broker is down; the > notificationlistener throws NPEs. > this test should have never been passed...there is a point where it drops a > database; and that command returns with an error - there are other things > which are intrestinglike create database on an existing db is sucess > somewhere - so it get posted to the msgbus. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[ https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18330: Attachment: HIVE-18330.01.patch > Fix TestMsgBusConnection - doesn't test tests the original intention > > > Key: HIVE-18330 > URL: https://issues.apache.org/jira/browse/HIVE-18330 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-18330.01.patch > > > If the msgbus usage is configured; and the ActiveMQ broker is down; the > notificationlistener throws NPEs. > this test should have never been passed...there is a point where it drops a > database; and that command returns with an error - there are other things > which are intrestinglike create database on an existing db is sucess > somewhere - so it get posted to the msgbus. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18108) in case basic stats are missing; rowcount estimation depends on the selected columns size
[
https://issues.apache.org/jira/browse/HIVE-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301373#comment-16301373
]
Hive QA commented on HIVE-18108:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903397/HIVE-18108.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 11540 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_view]
(batchId=110)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8360/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8360/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8360/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903397 - PreCommit-HIVE-Build
> in case basic stats are missing; rowcount estimation depends on the selected
> columns size
> -
>
> Key: HIVE-18108
> URL: https://issues.apache.org/jira/browse/HIVE-18108
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18108.01.patch, HIVE-18108.02.patch,
> HIVE-18108.02wip02.patch
>
>
> in case basicstats are not available (especially rowcount):
> {code}
> set hive.stats.autogather=false;
> create table t (a integer, b string);
> insert into t values (1,'asd1');
> insert into t values (2,'asd2');
> insert into t values (3,'asd3');
> insert into t values (4,'asd4');
> insert into t values (5,'asd5');
> explain select a,count(1) from t group by a;
> -- estimated to read 8 rows from table t
> explain select b,count(1) from t group by b;
> -- estimated: 1 rows
> explain select a,b,count(1) from t group by a,b;
> -- estimated: 1 rows
> {code}
> it may not depend on the actually selected column set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18330) Fix TestMsgBusConnection - doesn't test tests the original intention
[ https://issues.apache.org/jira/browse/HIVE-18330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-18330: --- Assignee: Zoltan Haindrich Description: If the msgbus usage is configured; and the ActiveMQ broker is down; the notificationlistener throws NPEs. this test should have never been passed...there is a point where it drops a database; and that command returns with an error - there are other things which are intrestinglike create database on an existing db is sucess somewhere - so it get posted to the msgbus. was: this test should have never been passed...there is a point where it drops a database; and that command returns with an error - there are other things which are intrestinglike create database on an existing db is sucess somewhere - so it get posted to the msgbus. > Fix TestMsgBusConnection - doesn't test tests the original intention > > > Key: HIVE-18330 > URL: https://issues.apache.org/jira/browse/HIVE-18330 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > > If the msgbus usage is configured; and the ActiveMQ broker is down; the > notificationlistener throws NPEs. > this test should have never been passed...there is a point where it drops a > database; and that command returns with an error - there are other things > which are intrestinglike create database on an existing db is sucess > somewhere - so it get posted to the msgbus. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-3628) Provide a way to use counters in Hive through UDF
[
https://issues.apache.org/jira/browse/HIVE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301355#comment-16301355
]
Hudson commented on HIVE-3628:
--
[
https://issues-test.apache.org/jira/browse/HIVE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265589#comment-16265589
]
Pankaj Khattar edited comment on HIVE-3628 at 12/22/17 12:54 PM:
-
Usage:
public class UDAFEvaluator extends GenericUDAFEvaluator{
@Override
*public* *void* iterate(*final* +AggregationBuffer+ agg, *final* Object[]
parameters) *throws* HiveException
{ MapredContext.get().getReporter().getCounter(...).increment(1); }
}
was (Author: pankajkhattar):
Usage:
public class UDAFEvaluator extends GenericUDAFEvaluator{
@Override
*public* *void* iterate(*final* +AggregationBuffer+ agg, *final* Object[]
parameters) *throws* HiveException
{
MapredContext._get_().getReporter().getCounter(...).increment(1);
}
}
--
This message was sent by Atlassian JIRA
(v7.6.0#76001)
> Provide a way to use counters in Hive through UDF
> -
>
> Key: HIVE-3628
> URL: https://issues.apache.org/jira/browse/HIVE-3628
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Viji
>Assignee: Navis
>Priority: Minor
> Labels: TODOC11
> Fix For: 0.11.0
>
> Attachments: HIVE-3628.D8007.1.patch, HIVE-3628.D8007.2.patch,
> HIVE-3628.D8007.3.patch, HIVE-3628.D8007.4.patch, HIVE-3628.D8007.5.patch,
> HIVE-3628.D8007.6.patch
>
>
> Currently it is not possible to generate counters through UDF. We should
> support this.
> Pig currently allows this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-3628) Provide a way to use counters in Hive through UDF
[
https://issues.apache.org/jira/browse/HIVE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301353#comment-16301353
]
Hudson commented on HIVE-3628:
--
[
https://issues-test.apache.org/jira/browse/HIVE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265589#comment-16265589
]
Pankaj Khattar commented on HIVE-3628:
--
Usage:
public class UDAFEvaluator extends GenericUDAFEvaluator{
@Override
*public* *void* iterate(*final* +AggregationBuffer+ agg, *final* Object[]
parameters) *throws* HiveException
{
MapredContext._get_().getReporter().getCounter(...).increment(1);
}
}
--
This message was sent by Atlassian JIRA
(v7.6.0#76001)
> Provide a way to use counters in Hive through UDF
> -
>
> Key: HIVE-3628
> URL: https://issues.apache.org/jira/browse/HIVE-3628
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Viji
>Assignee: Navis
>Priority: Minor
> Labels: TODOC11
> Fix For: 0.11.0
>
> Attachments: HIVE-3628.D8007.1.patch, HIVE-3628.D8007.2.patch,
> HIVE-3628.D8007.3.patch, HIVE-3628.D8007.4.patch, HIVE-3628.D8007.5.patch,
> HIVE-3628.D8007.6.patch
>
>
> Currently it is not possible to generate counters through UDF. We should
> support this.
> Pig currently allows this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-3628) Provide a way to use counters in Hive through UDF
[
https://issues-test.apache.org/jira/browse/HIVE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265589#comment-16265589
]
Pankaj Khattar edited comment on HIVE-3628 at 12/22/17 12:54 PM:
-
Usage:
public class UDAFEvaluator extends GenericUDAFEvaluator{
@Override
*public* *void* iterate(*final* +AggregationBuffer+ agg, *final* Object[]
parameters) *throws* HiveException
{ MapredContext.get().getReporter().getCounter(...).increment(1); }
}
was (Author: pankajkhattar):
Usage:
public class UDAFEvaluator extends GenericUDAFEvaluator{
@Override
*public* *void* iterate(*final* +AggregationBuffer+ agg, *final* Object[]
parameters) *throws* HiveException
{
MapredContext._get_().getReporter().getCounter(...).increment(1);
}
}
> Provide a way to use counters in Hive through UDF
> -
>
> Key: HIVE-3628
> URL: https://issues-test.apache.org/jira/browse/HIVE-3628
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Viji
>Assignee: Navis
>Priority: Minor
> Labels: TODOC11
> Fix For: 0.11.0
>
> Attachments: HIVE-3628.D8007.1.patch, HIVE-3628.D8007.2.patch,
> HIVE-3628.D8007.3.patch, HIVE-3628.D8007.4.patch, HIVE-3628.D8007.5.patch,
> HIVE-3628.D8007.6.patch
>
>
> Currently it is not possible to generate counters through UDF. We should
> support this.
> Pig currently allows this.
--
This message was sent by Atlassian JIRA
(v7.6.0#76001)
[jira] [Commented] (HIVE-3628) Provide a way to use counters in Hive through UDF
[
https://issues-test.apache.org/jira/browse/HIVE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265589#comment-16265589
]
Pankaj Khattar commented on HIVE-3628:
--
Usage:
public class UDAFEvaluator extends GenericUDAFEvaluator{
@Override
*public* *void* iterate(*final* +AggregationBuffer+ agg, *final* Object[]
parameters) *throws* HiveException
{
MapredContext._get_().getReporter().getCounter(...).increment(1);
}
}
> Provide a way to use counters in Hive through UDF
> -
>
> Key: HIVE-3628
> URL: https://issues-test.apache.org/jira/browse/HIVE-3628
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Viji
>Assignee: Navis
>Priority: Minor
> Labels: TODOC11
> Fix For: 0.11.0
>
> Attachments: HIVE-3628.D8007.1.patch, HIVE-3628.D8007.2.patch,
> HIVE-3628.D8007.3.patch, HIVE-3628.D8007.4.patch, HIVE-3628.D8007.5.patch,
> HIVE-3628.D8007.6.patch
>
>
> Currently it is not possible to generate counters through UDF. We should
> support this.
> Pig currently allows this.
--
This message was sent by Atlassian JIRA
(v7.6.0#76001)
[jira] [Commented] (HIVE-18108) in case basic stats are missing; rowcount estimation depends on the selected columns size
[
https://issues.apache.org/jira/browse/HIVE-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301312#comment-16301312
]
Hive QA commented on HIVE-18108:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 96 unchanged - 0 fixed
= 97 total (was 96) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 0e2dab9 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8360/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8360/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> in case basic stats are missing; rowcount estimation depends on the selected
> columns size
> -
>
> Key: HIVE-18108
> URL: https://issues.apache.org/jira/browse/HIVE-18108
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18108.01.patch, HIVE-18108.02.patch,
> HIVE-18108.02wip02.patch
>
>
> in case basicstats are not available (especially rowcount):
> {code}
> set hive.stats.autogather=false;
> create table t (a integer, b string);
> insert into t values (1,'asd1');
> insert into t values (2,'asd2');
> insert into t values (3,'asd3');
> insert into t values (4,'asd4');
> insert into t values (5,'asd5');
> explain select a,count(1) from t group by a;
> -- estimated to read 8 rows from table t
> explain select b,count(1) from t group by b;
> -- estimated: 1 rows
> explain select a,b,count(1) from t group by a,b;
> -- estimated: 1 rows
> {code}
> it may not depend on the actually selected column set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18108) in case basic stats are missing; rowcount estimation depends on the selected columns size
[
https://issues.apache.org/jira/browse/HIVE-18108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18108:
Attachment: HIVE-18108.02.patch
> in case basic stats are missing; rowcount estimation depends on the selected
> columns size
> -
>
> Key: HIVE-18108
> URL: https://issues.apache.org/jira/browse/HIVE-18108
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-18108.01.patch, HIVE-18108.02.patch,
> HIVE-18108.02wip02.patch
>
>
> in case basicstats are not available (especially rowcount):
> {code}
> set hive.stats.autogather=false;
> create table t (a integer, b string);
> insert into t values (1,'asd1');
> insert into t values (2,'asd2');
> insert into t values (3,'asd3');
> insert into t values (4,'asd4');
> insert into t values (5,'asd5');
> explain select a,count(1) from t group by a;
> -- estimated to read 8 rows from table t
> explain select b,count(1) from t group by b;
> -- estimated: 1 rows
> explain select a,b,count(1) from t group by a,b;
> -- estimated: 1 rows
> {code}
> it may not depend on the actually selected column set.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17434) Using "add jar " from viewFs always occurred hdfs mismatch error
[ https://issues.apache.org/jira/browse/HIVE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bang Xiao updated HIVE-17434: - Affects Version/s: 1.2.3 1.2.2 Target Version/s: 1.2.2, 1.2.1, 1.2.3 Fix Version/s: 1.2.3 > Using "add jar " from viewFs always occurred hdfs mismatch error > > > Key: HIVE-17434 > URL: https://issues.apache.org/jira/browse/HIVE-17434 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 1.2.2, 1.2.3 >Reporter: shenxianqiang >Assignee: Bang Xiao >Priority: Minor > Fix For: 1.2.1, 1.2.3 > > Attachments: HIVE-17434.2-branch-1.2.patch, HIVE-17434.patch > > > add jar viewfs://nsX//lib/common.jar > always occure mismatch error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17434) Using "add jar " from viewFs always occurred hdfs mismatch error
[ https://issues.apache.org/jira/browse/HIVE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bang Xiao updated HIVE-17434: - Attachment: HIVE-17434.2-branch-1.2.patch add branch tags for the patch > Using "add jar " from viewFs always occurred hdfs mismatch error > > > Key: HIVE-17434 > URL: https://issues.apache.org/jira/browse/HIVE-17434 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: shenxianqiang >Assignee: Bang Xiao >Priority: Minor > Fix For: 1.2.1 > > Attachments: HIVE-17434.2-branch-1.2.patch, HIVE-17434.patch > > > add jar viewfs://nsX//lib/common.jar > always occure mismatch error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17434) Using "add jar " from viewFs always occurred hdfs mismatch error
[ https://issues.apache.org/jira/browse/HIVE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bang Xiao updated HIVE-17434: - Status: Open (was: Patch Available) > Using "add jar " from viewFs always occurred hdfs mismatch error > > > Key: HIVE-17434 > URL: https://issues.apache.org/jira/browse/HIVE-17434 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: shenxianqiang >Assignee: Bang Xiao >Priority: Minor > Fix For: 1.2.1 > > Attachments: HIVE-17434.patch > > > add jar viewfs://nsX//lib/common.jar > always occure mismatch error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17434) Using "add jar " from viewFs always occurred hdfs mismatch error
[ https://issues.apache.org/jira/browse/HIVE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bang Xiao updated HIVE-17434: - Attachment: (was: HIVE-17434-1.patch) > Using "add jar " from viewFs always occurred hdfs mismatch error > > > Key: HIVE-17434 > URL: https://issues.apache.org/jira/browse/HIVE-17434 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: shenxianqiang >Assignee: Bang Xiao >Priority: Minor > Fix For: 1.2.1 > > Attachments: HIVE-17434.patch > > > add jar viewfs://nsX//lib/common.jar > always occure mismatch error -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18224) Introduce interface above driver
[ https://issues.apache.org/jira/browse/HIVE-18224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18224: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) pushed to master. Thank you Ashutosh for reviewing the changes! > Introduce interface above driver > > > Key: HIVE-18224 > URL: https://issues.apache.org/jira/browse/HIVE-18224 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 3.0.0 > > Attachments: HIVE-18224.01.patch, HIVE-18224.02.patch, > HIVE-18224.03.patch, HIVE-18224.04.patch > > > Add an interface above driver; and use it outside of ql. > The goal is to enable the overlaying of the Driver with some strategy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
