[jira] [Work logged] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?focusedWorklogId=286406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-286406
]
ASF GitHub Bot logged work on HIVE-21960:
-
Author: ASF GitHub Bot
Created on: 01/Aug/19 05:03
Start Date: 01/Aug/19 05:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #735: HIVE-21960 :
Avoid running stats updater and partition management task on a replicated table.
URL: https://github.com/apache/hive/pull/735#discussion_r309523387
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java
##
@@ -80,6 +82,16 @@ public Configuration getConf() {
return conf;
}
+ private static boolean partitionDiscoveryEnabled(Map params)
{
+return params != null &&
params.containsKey(DISCOVER_PARTITIONS_TBLPROPERTY) &&
+
params.get(DISCOVER_PARTITIONS_TBLPROPERTY).equalsIgnoreCase("true");
+ }
+
+ private static boolean tblBeingReplicatedInto(Map params) {
+return params != null &&
params.containsKey(ReplConst.REPL_TARGET_TABLE_PROPERTY) &&
+!params.get(ReplConst.REPL_TARGET_TABLE_PROPERTY).trim().isEmpty();
Review comment:
isEmpty() check looks redundant
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 286406)
Time Spent: 2h (was: 1h 50m)
> HMS tasks on replica
>
>
> Key: HIVE-21960
> URL: https://issues.apache.org/jira/browse/HIVE-21960
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch,
> HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch,
> HIVE-21960.06.patch, Replication and House keeping tasks.pdf
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> An HMS performs a number of housekeeping tasks. Assess whether
> # They are required to be performed in the replicated data
> # Performing those on replicated data causes any issues and how to fix those.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (HIVE-21960) HMS tasks on replica
[ https://issues.apache.org/jira/browse/HIVE-21960?focusedWorklogId=286404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-286404 ] ASF GitHub Bot logged work on HIVE-21960: - Author: ASF GitHub Bot Created on: 01/Aug/19 05:00 Start Date: 01/Aug/19 05:00 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #735: HIVE-21960 : Avoid running stats updater and partition management task on a replicated table. URL: https://github.com/apache/hive/pull/735#discussion_r309522942 ## File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/creation/CreateTableDesc.java ## @@ -27,6 +27,7 @@ import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.common.StatsSetupConst; +import org.apache.hadoop.hive.common.repl.ReplConst; Review comment: looks like a redundant import This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 286404) Time Spent: 1h 50m (was: 1h 40m) > HMS tasks on replica > > > Key: HIVE-21960 > URL: https://issues.apache.org/jira/browse/HIVE-21960 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch, > HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch, > HIVE-21960.06.patch, Replication and House keeping tasks.pdf > > Time Spent: 1h 50m > Remaining Estimate: 0h > > An HMS performs a number of housekeeping tasks. Assess whether > # They are required to be performed in the replicated data > # Performing those on replicated data causes any issues and how to fix those. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[
https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897768#comment-16897768
]
Hive QA commented on HIVE-22063:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 23s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-18227/patches/PreCommit-HIVE-Build-18227.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18227/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ranger Authorization in Hive based on object ownership - HMS code path
> --
>
> Key: HIVE-22063
> URL: https://issues.apache.org/jira/browse/HIVE-22063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch,
> HIVE-22063.3.patch
>
>
> This takes care of adding the owner and ownertype in the HMS code path
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897764#comment-16897764
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976375/HIVE-21637.54.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18226/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18226/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18226/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12976375/HIVE-21637.54.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976375 - PreCommit-HIVE-Build
> Synchronized metastore cache
>
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch,
> HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch,
> HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch,
> HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch,
> HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch,
> HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch,
> HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch,
> HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch,
> HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch,
> HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch,
> HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch,
> HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch,
> HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch,
> HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch,
> HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch,
> HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch,
> HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch,
> HIVE-21637.54.patch, HIVE-21637.6.patch, HIVE-21637.7.patch,
> HIVE-21637.8.patch, HIVE-21637.9.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is
> asynchronized and in HMS HA setting, we can only get eventual consistency. In
> this Jira, we try to make it synchronized.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897762#comment-16897762
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976375/HIVE-21637.54.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18225/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18225/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18225/
Messages:
{noformat}
This message was trimmed, see log for full details
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java:816
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java:716
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsResponse.java:354
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsResponse.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java:877
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java:351
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java:354
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClientCapabilities.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionRequest.java:814
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionRequest.java'
with conflicts.
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CreateTableRequest.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ExtendedTableInfo.java:529
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ExtendedTableInfo.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java:350
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java:794
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java:346
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java'
with conflicts.
error: patch failed:
[jira] [Commented] (HIVE-22066) Upgrade Apache parent POM to version 21
[
https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897760#comment-16897760
]
Hive QA commented on HIVE-22066:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976373/HIVE-22066.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 16600 tests
executed
*Failed tests:*
{noformat}
TestAMReporter - did not produce a TEST-*.xml file (likely timed out)
(batchId=361)
TestBuddyAllocator - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestBuddyAllocatorForceEvict - did not produce a TEST-*.xml file (likely timed
out) (batchId=360)
TestCacheAllocationsEvictionsCycles - did not produce a TEST-*.xml file (likely
timed out) (batchId=360)
TestContainerRunnerImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestEvictingPriorityBlockingQueue - did not produce a TEST-*.xml file (likely
timed out) (batchId=360)
TestFirstInFirstOutComparator - did not produce a TEST-*.xml file (likely timed
out) (batchId=361)
TestIncrementalObjectSizeEstimator - did not produce a TEST-*.xml file (likely
timed out) (batchId=360)
TestLlapDaemon - did not produce a TEST-*.xml file (likely timed out)
(batchId=361)
TestLlapDaemonExecutorMetrics - did not produce a TEST-*.xml file (likely timed
out) (batchId=360)
TestLlapDaemonProtocolServerImpl - did not produce a TEST-*.xml file (likely
timed out) (batchId=361)
TestLlapLoadGeneratorService - did not produce a TEST-*.xml file (likely timed
out) (batchId=360)
TestLlapRegistryService - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestLlapServiceCommandLine - did not produce a TEST-*.xml file (likely timed
out) (batchId=360)
TestLlapSignerImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestLlapStatusServiceCommandLine - did not produce a TEST-*.xml file (likely
timed out) (batchId=360)
TestLlapTokenChecker - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestLlapWebServices - did not produce a TEST-*.xml file (likely timed out)
(batchId=361)
TestLowLevelCacheImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestLowLevelLrfuCachePolicy - did not produce a TEST-*.xml file (likely timed
out) (batchId=360)
TestOrcMetadataCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestQueryIdentifier - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestShortestJobFirstComparator - did not produce a TEST-*.xml file (likely
timed out) (batchId=360)
TestTaskExecutorService - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestVectorDeserializeOrcWriter - did not produce a TEST-*.xml file (likely
timed out) (batchId=360)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=197)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18224/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18224/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976373 - PreCommit-HIVE-Build
> Upgrade Apache parent POM to version 21
> ---
>
> Key: HIVE-22066
> URL: https://issues.apache.org/jira/browse/HIVE-22066
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22066.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22066) Upgrade Apache parent POM to version 21
[
https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897746#comment-16897746
]
Hive QA commented on HIVE-22066:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18224/dev-support/hive-personality.sh
|
| git revision | master / 1fb0107 |
| Default Java | 1.8.0_111 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18224/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade Apache parent POM to version 21
> ---
>
> Key: HIVE-22066
> URL: https://issues.apache.org/jira/browse/HIVE-22066
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22066.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897738#comment-16897738 ] Sam An commented on HIVE-22063: --- 22063.3.patch is a dummy that contains only latest master commit. Just test to see if PreCommit test is flaky. > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.3.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch, > HIVE-22063.3.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[
https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897726#comment-16897726
]
Hive QA commented on HIVE-22045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976372/HIVE-22045.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18223/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18223/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12976372/HIVE-22045.7.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976372 - PreCommit-HIVE-Build
> HIVE-21711 introduced regression in data load
> -
>
> Key: HIVE-22045
> URL: https://issues.apache.org/jira/browse/HIVE-22045
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch,
> HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch,
> HIVE-22045.6.patch, HIVE-22045.7.patch
>
>
> Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV
> statements to avoid intermittent rename operation but keep INSERT etc
> statements do intermittent rename since otherwise final move by file
> operation is significantly slow for such statements.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[
https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897725#comment-16897725
]
Hive QA commented on HIVE-22045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976372/HIVE-22045.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18222/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18222/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18222/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12976372/HIVE-22045.7.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976372 - PreCommit-HIVE-Build
> HIVE-21711 introduced regression in data load
> -
>
> Key: HIVE-22045
> URL: https://issues.apache.org/jira/browse/HIVE-22045
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch,
> HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch,
> HIVE-22045.6.patch, HIVE-22045.7.patch
>
>
> Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV
> statements to avoid intermittent rename operation but keep INSERT etc
> statements do intermittent rename since otherwise final move by file
> operation is significantly slow for such statements.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[
https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897724#comment-16897724
]
Hive QA commented on HIVE-22045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976372/HIVE-22045.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16717 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18221/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18221/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18221/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976372 - PreCommit-HIVE-Build
> HIVE-21711 introduced regression in data load
> -
>
> Key: HIVE-22045
> URL: https://issues.apache.org/jira/browse/HIVE-22045
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch,
> HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch,
> HIVE-22045.6.patch, HIVE-22045.7.patch
>
>
> Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV
> statements to avoid intermittent rename operation but keep INSERT etc
> statements do intermittent rename since otherwise final move by file
> operation is significantly slow for such statements.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[
https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897709#comment-16897709
]
Hive QA commented on HIVE-22045:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 1 new + 665 unchanged - 0
fixed = 666 total (was 665) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18221/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18221/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18221/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HIVE-21711 introduced regression in data load
> -
>
> Key: HIVE-22045
> URL: https://issues.apache.org/jira/browse/HIVE-22045
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch,
> HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch,
> HIVE-22045.6.patch, HIVE-22045.7.patch
>
>
> Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV
> statements to avoid intermittent rename operation but keep INSERT etc
> statements do intermittent rename since otherwise final move by file
> operation is significantly slow for such statements.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[
https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897695#comment-16897695
]
Hive QA commented on HIVE-22063:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976357/HIVE-22063.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16717 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=360)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryNonDefaultCatalog
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18220/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18220/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18220/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976357 - PreCommit-HIVE-Build
> Ranger Authorization in Hive based on object ownership - HMS code path
> --
>
> Key: HIVE-22063
> URL: https://issues.apache.org/jira/browse/HIVE-22063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch
>
>
> This takes care of adding the owner and ownertype in the HMS code path
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[
https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897676#comment-16897676
]
Hive QA commented on HIVE-22063:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 9 unchanged - 1
fixed = 9 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18220/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18220/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ranger Authorization in Hive based on object ownership - HMS code path
> --
>
> Key: HIVE-22063
> URL: https://issues.apache.org/jira/browse/HIVE-22063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch
>
>
> This takes care of adding the owner and ownertype in the HMS code path
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897669#comment-16897669
]
Jesus Camacho Rodriguez commented on HIVE-22008:
+1
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-22008.1.patch, HIVE-22008.2.patch
>
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897649#comment-16897649
]
Hive QA commented on HIVE-22008:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976356/HIVE-22008.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16717 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18219/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18219/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18219/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976356 - PreCommit-HIVE-Build
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-22008.1.patch, HIVE-22008.2.patch
>
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21637: -- Attachment: HIVE-21637.54.patch > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.6.patch, HIVE-21637.7.patch, > HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897623#comment-16897623
]
Hive QA commented on HIVE-22008:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18219/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18219/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-22008.1.patch, HIVE-22008.2.patch
>
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
>
[jira] [Updated] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22066: --- Attachment: HIVE-22066.patch > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22066.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22066: -- > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22066 started by Jesus Camacho Rodriguez. -- > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22066: --- Status: Patch Available (was: In Progress) > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897614#comment-16897614
]
Hive QA commented on HIVE-20683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976353/HIVE-20683.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16718 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query1b]
(batchId=296)
org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1
(batchId=341)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18218/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18218/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18218/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976353 - PreCommit-HIVE-Build
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch,
> HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897609#comment-16897609
]
Hive QA commented on HIVE-20683:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18218/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql druid-handler . itests itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18218/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch,
> HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
>
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Attachment: HIVE-22045.7.patch > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, > HIVE-22045.6.patch, HIVE-22045.7.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Status: Open (was: Patch Available) > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, > HIVE-22045.6.patch, HIVE-22045.7.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Status: Patch Available (was: Open) > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, > HIVE-22045.6.patch, HIVE-22045.7.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897547#comment-16897547
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976352/HIVE-21637.53.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18217/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18217/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12976352/HIVE-21637.53.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976352 - PreCommit-HIVE-Build
> Synchronized metastore cache
>
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch,
> HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch,
> HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch,
> HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch,
> HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch,
> HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch,
> HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch,
> HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch,
> HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch,
> HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch,
> HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch,
> HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch,
> HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch,
> HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch,
> HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch,
> HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch,
> HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch,
> HIVE-21637.6.patch, HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is
> asynchronized and in HMS HA setting, we can only get eventual consistency. In
> this Jira, we try to make it synchronized.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897523#comment-16897523 ] Gopal V commented on HIVE-22045: LGTM - +1 > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, HIVE-22045.6.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Attachment: HIVE-22045.6.patch > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, HIVE-22045.6.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Status: Open (was: Patch Available) > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, HIVE-22045.6.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Status: Patch Available (was: Open) > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch, HIVE-22045.6.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897513#comment-16897513
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976352/HIVE-21637.53.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18216/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18216/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18216/
Messages:
{noformat}
This message was trimmed, see log for full details
error: a/ql/src/test/queries/clientpositive/perf/query49.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query5.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query50.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query51.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query52.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query53.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query54.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query55.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query56.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query57.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query58.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query59.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query6.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query60.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query61.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query63.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query64.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query65.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query66.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query67.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query68.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query69.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query7.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query70.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query71.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query72.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query73.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query74.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query75.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query76.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query77.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query78.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query79.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query8.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query80.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query81.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query82.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query83.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query84.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query85.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query86.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query87.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query88.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query89.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query9.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query90.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query91.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query92.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query93.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query94.q: does not exist in
index
error:
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Attachment: HIVE-22045.5.patch > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Status: Patch Available (was: Open) > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22045: --- Status: Open (was: Patch Available) > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch, HIVE-22045.5.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[ https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897485#comment-16897485 ] Vineet Garg commented on HIVE-22045: [~gopalv] [~jcamachorodriguez] I am uploading a new patch which should restore the behavior of non-blobstorage back to what it was in 3.1. With this patch new behavior as compared to old behavior is as follow: ||Blob-storage||Old behavior||New behavior|| |SELECT|Avoid intermittent rename and do final move file by file |*Avoid both intermittent rename* *as well as final move*| |CTAS/CMv|Avoid intermittent rename and do final move file by file|Avoid intermittent rename and do final move file by file| |INSERT|Avoid intermittent rename and do final move file by file|*Do intermittent rename and* *do final dir rename (*_FASTER)_| ||Non-blobstorage||Old behavior||New behavior|| |SELECT|Do intermittent rename and also final dir rename|*Avoid both intermittent rename* *as well as final move*| |CTAS/CMv|Do intermittent rename and also final dir rename|Do intermittent rename and also final dir rename| |INSERT|Do intermittent rename and also final dir rename|Do intermittent rename and also final dir rename| > HIVE-21711 introduced regression in data load > - > > Key: HIVE-22045 > URL: https://issues.apache.org/jira/browse/HIVE-22045 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch, > HIVE-22045.3.patch, HIVE-22045.4.patch > > > Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV > statements to avoid intermittent rename operation but keep INSERT etc > statements do intermittent rename since otherwise final move by file > operation is significantly slow for such statements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[
https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897476#comment-16897476
]
Hive QA commented on HIVE-22065:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976346/HIVE-22065.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16717 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18215/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18215/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18215/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976346 - PreCommit-HIVE-Build
> Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
> --
>
> Key: HIVE-22065
> URL: https://issues.apache.org/jira/browse/HIVE-22065
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-22065.1.patch, HIVE-22065.2.patch
>
>
> I addressed the following issues:
> - replace explicit generic definitions with diamond operator where applicable
> - fix field visibility
> - use lambda expression instead of instantiating a custom interface
> implementation
> - remove unnecessary null checks
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[ https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-22063: -- Attachment: HIVE-22063.2.patch > Ranger Authorization in Hive based on object ownership - HMS code path > -- > > Key: HIVE-22063 > URL: https://issues.apache.org/jira/browse/HIVE-22063 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Major > Attachments: HIVE-22063.1.patch, HIVE-22063.2.patch > > > This takes care of adding the owner and ownertype in the HMS code path -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[
https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897452#comment-16897452
]
Hive QA commented on HIVE-22065:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 11 unchanged - 8
fixed = 11 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18215/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18215/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
> --
>
> Key: HIVE-22065
> URL: https://issues.apache.org/jira/browse/HIVE-22065
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-22065.1.patch, HIVE-22065.2.patch
>
>
> I addressed the following issues:
> - replace explicit generic definitions with diamond operator where applicable
> - fix field visibility
> - use lambda expression instead of instantiating a custom interface
> implementation
> - remove unnecessary null checks
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897422#comment-16897422
]
Hive QA commented on HIVE-21960:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976345/HIVE-21960.06.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16721 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18214/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18214/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18214/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976345 - PreCommit-HIVE-Build
> HMS tasks on replica
>
>
> Key: HIVE-21960
> URL: https://issues.apache.org/jira/browse/HIVE-21960
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch,
> HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch,
> HIVE-21960.06.patch, Replication and House keeping tasks.pdf
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> An HMS performs a number of housekeeping tasks. Assess whether
> # They are required to be performed in the replicated data
> # Performing those on replicated data causes any issues and how to fix those.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-22008:
---
Status: Patch Available (was: Open)
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.4, 4.0.0
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-22008.1.patch, HIVE-22008.2.patch
>
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-22008:
---
Attachment: HIVE-22008.2.patch
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-22008.1.patch, HIVE-22008.2.patch
>
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897390#comment-16897390
]
Oleksiy Sayankin commented on HIVE-4605:
I've re-testes failed tests:
{code}
mvn test -Dtest=TestCliDriver
-Dqfile=orc_llap_counters1.q,orc_llap_counters.q,orc_ppd_basic.q,orc_ppd_schema_evol_3a.q,materialized_view_rewrite_4.q
{code}
The fix does not change their behaviour: they fail even without any changes in
master branch.
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897381#comment-16897381
]
Hive QA commented on HIVE-21960:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
23s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 50 unchanged - 1 fixed = 50 total (was 51) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 80 unchanged - 1 fixed
= 81 total (was 81) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 12 new + 237
unchanged - 12 fixed = 249 total (was 249) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18214/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18214/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18214/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18214/yetus.txt |
| Powered by |
[jira] [Updated] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20683:
Attachment: HIVE-20683.4.patch
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch,
> HIVE-20683.3.patch, HIVE-20683.4.patch, HIVE-20683.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897331#comment-16897331 ] Jason Dere commented on HIVE-21838: --- +1 > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.10.patch, HIVE-21838.11.patch, > HIVE-21838.12.patch, HIVE-21838.13.patch, HIVE-21838.13.patch, > HIVE-21838.14.patch, HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.5.patch, HIVE-21838.6.patch, > HIVE-21838.7.patch, HIVE-21838.8.patch, HIVE-21838.9.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897330#comment-16897330
]
Hive QA commented on HIVE-4605:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976343/HIVE-4605.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16717 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_4]
(batchId=161)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18213/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18213/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18213/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976343 - PreCommit-HIVE-Build
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21637: -- Attachment: HIVE-21637.53.patch > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.6.patch, HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897292#comment-16897292
]
Hive QA commented on HIVE-4605:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18213/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18213/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive job fails while closing reducer output - Unable to rename
> --
>
> Key: HIVE-4605
> URL: https://issues.apache.org/jira/browse/HIVE-4605
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0
> Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010
> x86_64 x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>Reporter: Link Qian
>Assignee: Oleksiy Sayankin
>Priority: Major
> Attachments: HIVE-4605.2.patch, HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> {code}
> create table iparea_analysis_orc (network int, ip string, )
> stored as ORC;
> {code}
> 2, insert table iparea_analysis_orc select network, ip, , the script
> success, but failed after add *OVERWRITE* keyword. the main error log list
> as here.
> {code}
> java.lang.RuntimeException: Hive Runtime Error while closing operators:
> Unable to rename output from:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
> to:
> hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
[jira] [Commented] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[
https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897278#comment-16897278
]
Hive QA commented on HIVE-22065:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976336/HIVE-22065.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16717 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=46)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=360)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18212/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18212/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976336 - PreCommit-HIVE-Build
> Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
> --
>
> Key: HIVE-22065
> URL: https://issues.apache.org/jira/browse/HIVE-22065
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-22065.1.patch, HIVE-22065.2.patch
>
>
> I addressed the following issues:
> - replace explicit generic definitions with diamond operator where applicable
> - fix field visibility
> - use lambda expression instead of instantiating a custom interface
> implementation
> - remove unnecessary null checks
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[
https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-4605:
---
Description:
1, create a table with ORC storage model
{code}
create table iparea_analysis_orc (network int, ip string, )
stored as ORC;
{code}
2, insert table iparea_analysis_orc select network, ip, , the script
success, but failed after add *OVERWRITE* keyword. the main error log list as
here.
{code}
java.lang.RuntimeException: Hive Runtime Error while closing operators: Unable
to rename output from:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
to:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
at
org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
output from:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
to:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
... 7 more
{code}
was:
1, create a table with ORC storage model
create table iparea_analysis_orc (network int, ip string, )
stored as ORC;
2, insert table iparea_analysis_orc select network, ip, , the script
success, but failed after add *OVERWRITE* keyword. the main error log list as
here.
ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable
to rename output from:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
to:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
at
org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
output from:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0
to:
hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
... 7 more
> Hive job fails while closing reducer output - Unable to rename
> --
>
[jira] [Updated] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22065: --- Attachment: HIVE-22065.2.patch > Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor > -- > > Key: HIVE-22065 > URL: https://issues.apache.org/jira/browse/HIVE-22065 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22065.1.patch, HIVE-22065.2.patch > > > I addressed the following issues: > - replace explicit generic definitions with diamond operator where applicable > - fix field visibility > - use lambda expression instead of instantiating a custom interface > implementation > - remove unnecessary null checks -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[
https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897233#comment-16897233
]
Hive QA commented on HIVE-22065:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 11 unchanged - 8 fixed
= 12 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18212/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18212/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18212/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
> --
>
> Key: HIVE-22065
> URL: https://issues.apache.org/jira/browse/HIVE-22065
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-22065.1.patch
>
>
> I addressed the following issues:
> - replace explicit generic definitions with diamond operator where applicable
> - fix field visibility
> - use lambda expression instead of instantiating a custom interface
> implementation
> - remove unnecessary null checks
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-21960) HMS tasks on replica
[ https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21960: -- Attachment: HIVE-21960.06.patch Status: Patch Available (was: In Progress) Attaching 05 as 06 patch again to trigger ptest. > HMS tasks on replica > > > Key: HIVE-21960 > URL: https://issues.apache.org/jira/browse/HIVE-21960 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch, > HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch, > HIVE-21960.06.patch, Replication and House keeping tasks.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > An HMS performs a number of housekeeping tasks. Assess whether > # They are required to be performed in the replicated data > # Performing those on replicated data causes any issues and how to fix those. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Pusztahazi updated HIVE-22002:
Description:
create table test_double(id int) partitioned by (dbtest double);
insert into test_double partition(dbtest) values (1,9.9); --> this works
insert into test_double partition(dbtest) values (1,10); --> this fails
But if we change it to
insert into test_double partition(dbtest) values (1, cast (10 as double)); it
succeeds
-> the problem is only seen when trying to insert a whole number i.e. 10, 10.0,
15, 14.0 etc. The issue is not seen when inserting a number with decimal values
other than 0. So insert of 10.1 goes though.
The underlying from the HMS is
{code}
2019-07-11T07:58:16,670 [pool-6-thread-196]: server.TThreadPoolServer
(TThreadPoolServer.java:run(297)) - occurred during processing of message.
java.lang.IndexOutOfBoundsException: Index: 0 at
java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
{code}
With {{hive.stats.column.autogather=false}}, this exception does not occur with
or without the explicit casting.
The issue stems from the fact that HS2 created a partition with value
{{dbtest=10}} for the table and the stats processor is attempting to add column
statistics for partition with value {{dbtest=10.0}}. Thus HMS
{{getPartitionsByNames}} cannot find the partition with that value and thus
fails to insert the stats. So while the failure initiates on HMS side, the
cause in the HS2 query planning.
It makes sense that turning off {{hive.stats.column.autogather}} resolves the
issue because there is no StatsTask in a query plan.
But {{SHOW PARTITIONS}} shows the partition as created while the query planner
is not including it any plan because of the absence of stats on the partition.
was:
create table test_double(id int) partitioned by (dbtest double);
insert into test_double partition(dbtest) values (1,9.9); --> this works
insert into test_double partition(dbtest) values (1,10); --> this fails
But if we change it to
insert into test_double partition(dbtest) values (1, cast (10 as double)); it
succeeds
-> the problem is only seen when trying to insert a whole number i.e. 10, 10.0,
15, 14.0 etc. The issue is not seen when inserting a number with decimal values
other than 0. So insert of 10.1 goes though.
The underlying exception from the HMS is
{code}
2019-07-11T07:58:16,670 ERROR [pool-6-thread-196]: server.TThreadPoolServer
(TThreadPoolServer.java:run(297)) - Error occurred during processing of
message. java.lang.IndexOutOfBoundsException: Index: 0 at
java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
{code}
With {{hive.stats.column.autogather=false}}, this exception does not occur with
or without the explicit casting.
The issue stems from the fact that HS2 created a partition with value
{{dbtest=10}} for the table and the stats processor is attempting to add column
statistics for partition with value {{dbtest=10.0}}. Thus HMS
{{getPartitionsByNames}} cannot find the partition with that value and thus
fails to insert the stats. So while the failure initiates on HMS side, the
cause in the HS2 query planning.
It makes sense that turning off {{hive.stats.column.autogather}} resolves the
issue because there is no StatsTask in a query plan.
But {{SHOW PARTITIONS}} shows the partition as created while the query planner
is not including it any plan because of the absence of stats on the partition.
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But
[jira] [Updated] (HIVE-21960) HMS tasks on replica
[ https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21960: -- Status: In Progress (was: Patch Available) > HMS tasks on replica > > > Key: HIVE-21960 > URL: https://issues.apache.org/jira/browse/HIVE-21960 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch, > HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch, Replication > and House keeping tasks.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > An HMS performs a number of housekeeping tasks. Assess whether > # They are required to be performed in the replicated data > # Performing those on replicated data causes any issues and how to fix those. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-4605: --- Status: Patch Available (was: In Progress) > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 2.3.0, 0.13.1, 0.13.0, 0.12.0, 0.11.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable > to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897203#comment-16897203 ] Oleksiy Sayankin commented on HIVE-4605: Re-based the origin patch to see Junit tests results. > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable > to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin updated HIVE-4605: --- Attachment: HIVE-4605.2.patch > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.2.patch, HIVE-4605.patch > > > 1, create a table with ORC storage model > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable > to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897196#comment-16897196 ] Naveen Gangam commented on HIVE-21838: -- [~jdere] Tests are clean. is this a go? Have you had a chance to look at the latest patch? (nothing else has changed besides the suggestions you made). Thanks > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-21838.10.patch, HIVE-21838.11.patch, > HIVE-21838.12.patch, HIVE-21838.13.patch, HIVE-21838.13.patch, > HIVE-21838.14.patch, HIVE-21838.2.patch, HIVE-21838.3.patch, > HIVE-21838.4.patch, HIVE-21838.5.patch, HIVE-21838.6.patch, > HIVE-21838.7.patch, HIVE-21838.8.patch, HIVE-21838.9.patch, HIVE-21838.patch > > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksiy Sayankin reassigned HIVE-4605: -- Assignee: Oleksiy Sayankin (was: Brock Noland) > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.patch > > > 1, create a table with ORC storage model > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable > to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
[ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-4605 started by Oleksiy Sayankin. -- > Hive job fails while closing reducer output - Unable to rename > -- > > Key: HIVE-4605 > URL: https://issues.apache.org/jira/browse/HIVE-4605 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1, 2.3.0 > Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 > x86_64 x86_64 x86_64 GNU/Linux > Hadoop 1.1.2 >Reporter: Link Qian >Assignee: Oleksiy Sayankin >Priority: Major > Attachments: HIVE-4605.patch > > > 1, create a table with ORC storage model > create table iparea_analysis_orc (network int, ip string, ) > stored as ORC; > 2, insert table iparea_analysis_orc select network, ip, , the script > success, but failed after add *OVERWRITE* keyword. the main error log list > as here. > ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable > to rename output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename > output from: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-1/_tmp.00_0 > to: > hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-1/00_0 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597) > at > org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309) > ... 7 more -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22065: --- Status: Patch Available (was: In Progress) > Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor > -- > > Key: HIVE-22065 > URL: https://issues.apache.org/jira/browse/HIVE-22065 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22065.1.patch > > > I addressed the following issues: > - replace explicit generic definitions with diamond operator where applicable > - fix field visibility > - use lambda expression instead of instantiating a custom interface > implementation > - remove unnecessary null checks -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22065: --- Attachment: HIVE-22065.1.patch > Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor > -- > > Key: HIVE-22065 > URL: https://issues.apache.org/jira/browse/HIVE-22065 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22065.1.patch > > > I addressed the following issues: > - replace explicit generic definitions with diamond operator where applicable > - fix field visibility > - use lambda expression instead of instantiating a custom interface > implementation > - remove unnecessary null checks -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22065 started by Ivan Suller. -- > Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor > -- > > Key: HIVE-22065 > URL: https://issues.apache.org/jira/browse/HIVE-22065 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > > I addressed the following issues: > - replace explicit generic definitions with diamond operator where applicable > - fix field visibility > - use lambda expression instead of instantiating a custom interface > implementation > - remove unnecessary null checks -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22065) Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor
[ https://issues.apache.org/jira/browse/HIVE-22065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller reassigned HIVE-22065: -- Assignee: Ivan Suller > Code cleanup around org.apache.hadoop.hive.ql.exec.tez.RecordProcessor > -- > > Key: HIVE-22065 > URL: https://issues.apache.org/jira/browse/HIVE-22065 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > > I addressed the following issues: > - replace explicit generic definitions with diamond operator where applicable > - fix field visibility > - use lambda expression instead of instantiating a custom interface > implementation > - remove unnecessary null checks -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22060) Replacing "catch Throwable" with a more restricted exception class
[ https://issues.apache.org/jira/browse/HIVE-22060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897132#comment-16897132 ] Peter Vary commented on HIVE-22060: --- [~isuller]: If done like this, I am sold on this idea :) > Replacing "catch Throwable" with a more restricted exception class > -- > > Key: HIVE-22060 > URL: https://issues.apache.org/jira/browse/HIVE-22060 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > > Catching Throwable considered unsafe in Java. A Throwable can be any Error > and those are JVM errors after the state of the JVM is not guaranteed thus > the cleanest way to "handle" the error is to let it kill the current thread. > I ran a quick scan and found almost 400 "catch Throwable" in the current > codebase. I opened this ticket as a conversation starter to: > - discuss if we want to eliminate this issue > - if we want to do it what's the best way to do it -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897130#comment-16897130
]
Hive QA commented on HIVE-21960:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976325/HIVE-21960.05.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16721 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18211/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18211/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976325 - PreCommit-HIVE-Build
> HMS tasks on replica
>
>
> Key: HIVE-21960
> URL: https://issues.apache.org/jira/browse/HIVE-21960
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch,
> HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch, Replication
> and House keeping tasks.pdf
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> An HMS performs a number of housekeeping tasks. Assess whether
> # They are required to be performed in the replicated data
> # Performing those on replicated data causes any issues and how to fix those.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22060) Replacing "catch Throwable" with a more restricted exception class
[
https://issues.apache.org/jira/browse/HIVE-22060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897128#comment-16897128
]
Ivan Suller commented on HIVE-22060:
[~pvary] There's almost always a way to avoid catching Throwable. For example
I'd write the above issue from this:
{code:java}
try {
...
} catch (Exception e) {
LOG.error("Caught exception while trying to compact " + ci +
". Marking failed to avoid repeated failures, " +
StringUtils.stringifyException(e));
msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
msc.abortTxns(Collections.singletonList(compactorTxnId));
}
{code}
to this:
{code:java}
boolean successfulCompaction = false;
try {
...
successfulCompaction = true;
} finally {
if (!successfulCompaction) {
msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
msc.abortTxns(Collections.singletonList(compactorTxnId));
}
} catch (Exception e) {
LOG.error("Caught exception while trying to compact " + ci +
". Marking failed to avoid repeated failures, " +
StringUtils.stringifyException(e));
}
{code}
> Replacing "catch Throwable" with a more restricted exception class
> --
>
> Key: HIVE-22060
> URL: https://issues.apache.org/jira/browse/HIVE-22060
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Major
>
> Catching Throwable considered unsafe in Java. A Throwable can be any Error
> and those are JVM errors after the state of the JVM is not guaranteed thus
> the cleanest way to "handle" the error is to let it kill the current thread.
> I ran a quick scan and found almost 400 "catch Throwable" in the current
> codebase. I opened this ticket as a conversation starter to:
> - discuss if we want to eliminate this issue
> - if we want to do it what's the best way to do it
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897106#comment-16897106
]
Hive QA commented on HIVE-21960:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
27s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 50 unchanged - 1 fixed = 50 total (was 51) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 80 unchanged - 1 fixed
= 81 total (was 81) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 12 new + 237
unchanged - 12 fixed = 249 total (was 249) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18211/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18211/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18211/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18211/yetus.txt |
| Powered by |
[jira] [Commented] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897102#comment-16897102
]
bencao commented on HIVE-22002:
---
Is this exception caused by this sql?
sql ->>> insert into test_double partition(dbtest) values (1, cast (10
as double));
!image-2019-07-31-20-02-38-069.png!
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying exception from the HMS is
> {code}
> 2019-07-11T07:58:16,670 ERROR [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - Error occurred during processing of
> message. java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22060) Replacing "catch Throwable" with a more restricted exception class
[ https://issues.apache.org/jira/browse/HIVE-22060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897100#comment-16897100 ] Peter Vary commented on HIVE-22060: --- Interesting stuff. I introduced one myself: See: HIVE-21402. I think there are cases where it makes sense to catch Throwable. > Replacing "catch Throwable" with a more restricted exception class > -- > > Key: HIVE-22060 > URL: https://issues.apache.org/jira/browse/HIVE-22060 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Major > > Catching Throwable considered unsafe in Java. A Throwable can be any Error > and those are JVM errors after the state of the JVM is not guaranteed thus > the cleanest way to "handle" the error is to let it kill the current thread. > I ran a quick scan and found almost 400 "catch Throwable" in the current > codebase. I opened this ticket as a conversation starter to: > - discuss if we want to eliminate this issue > - if we want to do it what's the best way to do it -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bencao updated HIVE-22002:
--
Attachment: image-2019-07-31-20-02-38-069.png
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
> Attachments: image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying exception from the HMS is
> {code}
> 2019-07-11T07:58:16,670 ERROR [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - Error occurred during processing of
> message. java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (HIVE-22002) Insert into table partition fails partially with stats.autogather is on.
[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bencao reassigned HIVE-22002:
-
Assignee: bencao
> Insert into table partition fails partially with stats.autogather is on.
>
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: bencao
>Priority: Major
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying exception from the HMS is
> {code}
> 2019-07-11T07:58:16,670 ERROR [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - Error occurred during processing of
> message. java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22053) Function name is not normalized when creating function
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897084#comment-16897084 ] Rui Li commented on HIVE-22053: --- Thanks [~pvary] :) > Function name is not normalized when creating function > -- > > Key: HIVE-22053 > URL: https://issues.apache.org/jira/browse/HIVE-22053 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22053.1.patch > > > If a function is created with a name containing upper case characters, we get > NoSuchObjectException when trying to get that function. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22053) Function name is not normalized when creating function
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897069#comment-16897069 ] Peter Vary commented on HIVE-22053: --- [~lirui]: Was on PTO. +1 after the event :D > Function name is not normalized when creating function > -- > > Key: HIVE-22053 > URL: https://issues.apache.org/jira/browse/HIVE-22053 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22053.1.patch > > > If a function is created with a name containing upper case characters, we get > NoSuchObjectException when trying to get that function. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21960) HMS tasks on replica
[ https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21960: -- Attachment: HIVE-21960.05.patch Status: Patch Available (was: In Progress) Fixed failures in the previous ptest run. > HMS tasks on replica > > > Key: HIVE-21960 > URL: https://issues.apache.org/jira/browse/HIVE-21960 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch, > HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch, Replication > and House keeping tasks.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > An HMS performs a number of housekeeping tasks. Assess whether > # They are required to be performed in the replicated data > # Performing those on replicated data causes any issues and how to fix those. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21960) HMS tasks on replica
[ https://issues.apache.org/jira/browse/HIVE-21960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21960: -- Status: In Progress (was: Patch Available) > HMS tasks on replica > > > Key: HIVE-21960 > URL: https://issues.apache.org/jira/browse/HIVE-21960 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch, > HIVE-21960.03.patch, HIVE-21960.04.patch, HIVE-21960.05.patch, Replication > and House keeping tasks.pdf > > Time Spent: 1h 40m > Remaining Estimate: 0h > > An HMS performs a number of housekeeping tasks. Assess whether > # They are required to be performed in the replicated data > # Performing those on replicated data causes any issues and how to fix those. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[
https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897052#comment-16897052
]
Hive QA commented on HIVE-21838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976288/HIVE-21838.14.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16718 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18210/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18210/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976288 - PreCommit-HIVE-Build
> Hive Metastore Translation: Add API call to tell client why table has limited
> access
>
>
> Key: HIVE-21838
> URL: https://issues.apache.org/jira/browse/HIVE-21838
> Project: Hive
> Issue Type: Sub-task
>Reporter: Yongzhi Chen
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-21838.10.patch, HIVE-21838.11.patch,
> HIVE-21838.12.patch, HIVE-21838.13.patch, HIVE-21838.13.patch,
> HIVE-21838.14.patch, HIVE-21838.2.patch, HIVE-21838.3.patch,
> HIVE-21838.4.patch, HIVE-21838.5.patch, HIVE-21838.6.patch,
> HIVE-21838.7.patch, HIVE-21838.8.patch, HIVE-21838.9.patch, HIVE-21838.patch
>
>
> When a table access type is Read-only or None, we need a way to tell clients
> why.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[
https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897040#comment-16897040
]
Hive QA commented on HIVE-21838:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
31s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 19 new + 389 unchanged - 28 fixed = 408 total (was 417) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
28s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 100 new + 1001 unchanged - 35 fixed = 1101 total (was 1036) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 20 new + 142
unchanged - 13 fixed = 162 total (was 155) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
25s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 179 unchanged - 0 fixed = 180 total (was 179) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
18s{color} | {color:red} standalone-metastore_metastore-common generated 2 new
+ 47 unchanged - 0 fixed = 49 total (was 47) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Private method
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(RawStore,
Table, EnvironmentContext, List, List, List, List, List, List, List, String)
is never called At HiveMetaStore.java:List, List, List, List, List, List,
List, String) is never called At HiveMetaStore.java:[lines 1960-1980] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Work logged] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?focusedWorklogId=285651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-285651
]
ASF GitHub Bot logged work on HIVE-21960:
-
Author: ASF GitHub Bot
Created on: 31/Jul/19 10:36
Start Date: 31/Jul/19 10:36
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #735: HIVE-21960 :
Avoid running stats updater and partition management task on a replicated table.
URL: https://github.com/apache/hive/pull/735#discussion_r309150433
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestPartitionManagement.java
##
@@ -563,6 +564,94 @@ public void testPartitionDiscoverySkipInvalidPath()
throws TException, IOExcepti
assertEquals(4, partitions.size());
}
+ @Test
+ public void testNoPartitionDiscoveryForReplTable() throws Exception {
+String dbName = "db_repl1";
+String tableName = "tbl_repl1";
+Map colMap = buildAllColumns();
+List partKeys = Lists.newArrayList("state", "dt");
+List partKeyTypes = Lists.newArrayList("string", "date");
+List> partVals = Lists.newArrayList(
+Lists.newArrayList("__HIVE_DEFAULT_PARTITION__", "1990-01-01"),
+Lists.newArrayList("CA", "1986-04-28"),
+Lists.newArrayList("MN", "2018-11-31"));
+createMetadata(DEFAULT_CATALOG_NAME, dbName, tableName, partKeys,
partKeyTypes, partVals, colMap, false);
+Table table = client.getTable(dbName, tableName);
+List partitions = client.listPartitions(dbName, tableName,
(short) -1);
+assertEquals(3, partitions.size());
+String tableLocation = table.getSd().getLocation();
+URI location = URI.create(tableLocation);
+Path tablePath = new Path(location);
+FileSystem fs = FileSystem.get(location, conf);
+Path newPart1 = new Path(tablePath, "state=WA/dt=2018-12-01");
+Path newPart2 = new Path(tablePath, "state=UT/dt=2018-12-02");
+fs.mkdirs(newPart1);
+fs.mkdirs(newPart2);
+assertEquals(5, fs.listStatus(tablePath).length);
+partitions = client.listPartitions(dbName, tableName, (short) -1);
+assertEquals(3, partitions.size());
+
+// table property is set to true, but the table is marked as replication
target. The new
+// partitions should not be created
+
table.getParameters().put(PartitionManagementTask.DISCOVER_PARTITIONS_TBLPROPERTY,
"true");
+table.getParameters().put(ReplConst.REPL_TARGET_PROPERTY, "1");
+client.alter_table(dbName, tableName, table);
+runPartitionManagementTask(conf);
+partitions = client.listPartitions(dbName, tableName, (short) -1);
+assertEquals(3, partitions.size());
+
+// change table type to external, delete a partition directory and make
sure partition discovery works
+table.getParameters().put("EXTERNAL", "true");
+table.setTableType(TableType.EXTERNAL_TABLE.name());
+client.alter_table(dbName, tableName, table);
+// Delete location of one of the partitions. The partition discovery task
should not drop
+// that partition.
+boolean deleted = fs.delete((new
Path(URI.create(partitions.get(0).getSd().getLocation(.getParent(),
+true);
+assertTrue(deleted);
+assertEquals(4, fs.listStatus(tablePath).length);
+runPartitionManagementTask(conf);
+partitions = client.listPartitions(dbName, tableName, (short) -1);
+assertEquals(3, partitions.size());
+ }
+
+ @Test
+ public void testNoPartitionRetentionForReplTarget() throws TException,
InterruptedException {
+String dbName = "db_repl2";
+String tableName = "tbl_repl2";
+Map colMap = buildAllColumns();
+List partKeys = Lists.newArrayList("state", "dt");
+List partKeyTypes = Lists.newArrayList("string", "date");
+List> partVals = Lists.newArrayList(
+Lists.newArrayList("__HIVE_DEFAULT_PARTITION__", "1990-01-01"),
+Lists.newArrayList("CA", "1986-04-28"),
+Lists.newArrayList("MN", "2018-11-31"));
+// Check for the existence of partitions 10 seconds after the partition
retention period has
+// elapsed. Gives enough time for the partition retention task to work.
+long partitionRetentionPeriodMs = 2;
+long waitingPeriodForTest = partitionRetentionPeriodMs + 10 * 1000;
+createMetadata(DEFAULT_CATALOG_NAME, dbName, tableName, partKeys,
partKeyTypes, partVals, colMap, false);
+Table table = client.getTable(dbName, tableName);
+List partitions = client.listPartitions(dbName, tableName,
(short) -1);
+assertEquals(3, partitions.size());
+
+
table.getParameters().put(PartitionManagementTask.DISCOVER_PARTITIONS_TBLPROPERTY,
"true");
+
table.getParameters().put(PartitionManagementTask.PARTITION_RETENTION_PERIOD_TBLPROPERTY,
+partitionRetentionPeriodMs + "ms");
+
[jira] [Work logged] (HIVE-21960) HMS tasks on replica
[
https://issues.apache.org/jira/browse/HIVE-21960?focusedWorklogId=285649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-285649
]
ASF GitHub Bot logged work on HIVE-21960:
-
Author: ASF GitHub Bot
Created on: 31/Jul/19 10:35
Start Date: 31/Jul/19 10:35
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #735: HIVE-21960 :
Avoid running stats updater and partition management task on a replicated table.
URL: https://github.com/apache/hive/pull/735#discussion_r309150305
##
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##
@@ -220,6 +221,16 @@ private void stopWorkers() {
String skipParam =
table.getParameters().get(SKIP_STATS_AUTOUPDATE_PROPERTY);
if ("true".equalsIgnoreCase(skipParam)) return null;
+// If the table is being replicated into,
+// 1. the stats are also replicated from the source, so we don't need
those to be calculated
+//on the target again
+// 2. updating stats requires a writeId to be created. Hence writeIds on
source and target
+//can get out of sync when stats are updated. That can cause
consistency issues.
+String replTrgtParam =
table.getParameters().get(ReplConst.REPL_TARGET_PROPERTY);
+if (replTrgtParam != null && !replTrgtParam.isEmpty()) {
+ return null;
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 285649)
Time Spent: 1.5h (was: 1h 20m)
> HMS tasks on replica
>
>
> Key: HIVE-21960
> URL: https://issues.apache.org/jira/browse/HIVE-21960
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, repl
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21960.01.patch, HIVE-21960.02.patch,
> HIVE-21960.03.patch, HIVE-21960.04.patch, Replication and House keeping
> tasks.pdf
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> An HMS performs a number of housekeeping tasks. Assess whether
> # They are required to be performed in the replicated data
> # Performing those on replicated data causes any issues and how to fix those.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896996#comment-16896996
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976263/HIVE-21637.52.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18209/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18209/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18209/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12976263/HIVE-21637.52.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976263 - PreCommit-HIVE-Build
> Synchronized metastore cache
>
>
> Key: HIVE-21637
> URL: https://issues.apache.org/jira/browse/HIVE-21637
> Project: Hive
> Issue Type: New Feature
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch,
> HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch,
> HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch,
> HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch,
> HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch,
> HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch,
> HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch,
> HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch,
> HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch,
> HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch,
> HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch,
> HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch,
> HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch,
> HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch,
> HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch,
> HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch,
> HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.6.patch,
> HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch
>
>
> Currently, HMS has a cache implemented by CachedStore. The cache is
> asynchronized and in HMS HA setting, we can only get eventual consistency. In
> this Jira, we try to make it synchronized.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896991#comment-16896991
]
Hive QA commented on HIVE-20683:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976272/HIVE-20683.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 16718 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_semijoin_reduction_all_types]
(batchId=197)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=110)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query1b]
(batchId=296)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18208/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18208/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18208/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976272 - PreCommit-HIVE-Build
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch,
> HIVE-20683.3.patch, HIVE-20683.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
> https://github.com/apache/incubator-druid/pull/6222)
> Implementation details -
> # Hive generates and passes the filters as part of 'filterExpr' in TableScan.
> # DruidQueryBasedRecordReader gets this filter passed as part of the conf.
> # During execution phase, before sending the query to druid in
> DruidQueryBasedRecordReader we will deserialize this filter, translate it
> into a DruidDimFilter and add it to existing DruidQuery. Tez executor
> already ensures that when we start reading results from the record reader,
> all the dynamic values are initialized.
> # Explaining a druid query also prints the query sent to druid as
> {{druid.json.query}}. We also need to make sure to update the druid query
> with the filters. During explain we do not have the actual values for the
> dynamic values, so instead of values we will print the dynamic expression
> itself as part of druid query.
> Note:- This work needs druid to be updated to version 0.13.0
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-20683) Add the Ability to push Dynamic Between and Bloom filters to Druid
[
https://issues.apache.org/jira/browse/HIVE-20683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896985#comment-16896985
]
Hive QA commented on HIVE-20683:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18208/dev-support/hive-personality.sh
|
| git revision | master / dd190d9 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql druid-handler . itests itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18208/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add the Ability to push Dynamic Between and Bloom filters to Druid
> --
>
> Key: HIVE-20683
> URL: https://issues.apache.org/jira/browse/HIVE-20683
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20683.1.patch, HIVE-20683.2.patch,
> HIVE-20683.3.patch, HIVE-20683.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For optimizing joins, Hive generates BETWEEN filter with min-max and BLOOM
> filter for filtering one side of semi-join.
> Druid 0.13.0 will have support for Bloom filters (Added via
>
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896899#comment-16896899
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976263/HIVE-21637.52.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18207/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18207/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18207/
Messages:
{noformat}
This message was trimmed, see log for full details
error: a/ql/src/test/queries/clientpositive/perf/query49.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query5.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query50.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query51.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query52.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query53.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query54.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query55.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query56.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query57.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query58.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query59.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query6.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query60.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query61.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query63.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query64.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query65.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query66.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query67.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query68.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query69.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query7.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query70.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query71.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query72.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query73.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query74.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query75.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query76.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query77.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query78.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query79.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query8.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query80.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query81.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query82.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query83.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query84.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query85.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query86.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query87.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query88.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query89.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query9.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query90.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query91.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query92.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query93.q: does not exist in
index
error: a/ql/src/test/queries/clientpositive/perf/query94.q: does not exist in
index
error:
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[
https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896896#comment-16896896
]
Hive QA commented on HIVE-22063:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976249/HIVE-22063.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16715 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_diff_fs]
(batchId=192)
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=360)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18206/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18206/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18206/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976249 - PreCommit-HIVE-Build
> Ranger Authorization in Hive based on object ownership - HMS code path
> --
>
> Key: HIVE-22063
> URL: https://issues.apache.org/jira/browse/HIVE-22063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22063.1.patch
>
>
> This takes care of adding the owner and ownertype in the HMS code path
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22053) Function name is not normalized when creating function
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-22053: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Guess PR is not mandatory. Pushed to master. Thanks Xuefu for the review! > Function name is not normalized when creating function > -- > > Key: HIVE-22053 > URL: https://issues.apache.org/jira/browse/HIVE-22053 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22053.1.patch > > > If a function is created with a name containing upper case characters, we get > NoSuchObjectException when trying to get that function. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22063) Ranger Authorization in Hive based on object ownership - HMS code path
[
https://issues.apache.org/jira/browse/HIVE-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896852#comment-16896852
]
Hive QA commented on HIVE-22063:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 1 new + 9 unchanged - 1 fixed
= 10 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18206/dev-support/hive-personality.sh
|
| git revision | master / b8afcc3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18206/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18206/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Ranger Authorization in Hive based on object ownership - HMS code path
> --
>
> Key: HIVE-22063
> URL: https://issues.apache.org/jira/browse/HIVE-22063
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Sam An
>Assignee: Sam An
>Priority: Major
> Attachments: HIVE-22063.1.patch
>
>
> This takes care of adding the owner and ownertype in the HMS code path
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22045) HIVE-21711 introduced regression in data load
[
https://issues.apache.org/jira/browse/HIVE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896816#comment-16896816
]
Hive QA commented on HIVE-22045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976254/HIVE-22045.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16715 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18205/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18205/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18205/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976254 - PreCommit-HIVE-Build
> HIVE-21711 introduced regression in data load
> -
>
> Key: HIVE-22045
> URL: https://issues.apache.org/jira/browse/HIVE-22045
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22045.1.patch, HIVE-22045.2.patch,
> HIVE-22045.3.patch, HIVE-22045.4.patch
>
>
> Better fix for HIVE-21711 is to specialize the handling for CTAS/Create MV
> statements to avoid intermittent rename operation but keep INSERT etc
> statements do intermittent rename since otherwise final move by file
> operation is significantly slow for such statements.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-22008:
---
Attachment: HIVE-22008.1.patch
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-22008.1.patch
>
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896804#comment-16896804
]
Gopal V commented on HIVE-22008:
Interestingly, this is working when it hits the ORC fast-path.
{code}
CREATE TEMPORARY TABLE SplitLines(`id` string) STORED AS ORC;
INSERT INTO SplitLines SELECT 'withdraw\ncash';
SELECT * FROM SplitLines WHERE `id` LIKE '%withdraw%cash' ORDER BY id;
SELECT count(*) FROM SplitLines WHERE `id` LIKE '%withdraw%cash';
{code}
that scenario works fine, which means this is a simple enough fix for Parquet
with a simple SEL.
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-22008:
---
Affects Version/s: 4.0.0
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0, 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (HIVE-22008) Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n) character in data
[
https://issues.apache.org/jira/browse/HIVE-22008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-22008:
--
Assignee: Gopal V
> Apache Hive 2.3.4 - Issue with combination of Like operator & newline (\n)
> character in data
> -
>
> Key: HIVE-22008
> URL: https://issues.apache.org/jira/browse/HIVE-22008
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.4
>Reporter: Shankar
>Assignee: Gopal V
>Priority: Major
>
> I am facing some issues while using *Like* operator & *newline* (\n)
> character. Below is the in details description :
>
>
> {color:#263238}*-- Hive
> Queries *{color}
> {color:#263238} – consider these are the reproduce steps.{color}
>
> {color:#263238}create table default.withdraw({color}
> {color:#263238}id string{color}
> {color:#263238}) stored as parquet;{color}
> {color:#263238} {color}
>
> {color:#263238}
> *insert into default.withdraw select 'withdraw\ncash';*{color}
> -- note here, added '\n' character
>
> *{color:#263238}--1) result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.078 seconds, Fetched: *1 row(s)*{color}
>
> *--2)* *{color:#263238}result = {color}{color:#cc}wrong{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%withdraw%cash';{color}
> {color:#263238}OK{color}
> {color:#263238}Time taken: 0.066 seconds{color}
>
> *--3)* *{color:#263238}result = {color}{color:#6aa84f}success{color}*
> {color:#263238}hive> select * from default.withdraw where id like
> '%cash%';{color}
> {color:#263238}OK{color}
> {color:#263238}withdraw{color}
> {color:#263238}cash{color}
> {color:#263238}Time taken: 0.086 seconds, Fetched: 1 row(s){color}
>
>
>
>
> {color:#263238}*-- Presto
> Queries -*{color}
> {color:#263238}FYI - Presto (v0.221) is using above table meta store. We
> tested above queries on presto too. {color}
> {color:#263238} {color}
>
> *--1)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--2)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%withdraw%cash';
> id
> --
> withdraw
> cash
> (1 row)
>
> *--3)* *{color:#263238}result ={color}* *{color:#6aa84f}success{color}*
> presto> select * from default.withdraw where id like '%cash%';
> id
> --
> withdraw
> cash
> (1 row){color:#263238}
> {color}
>
>
> *--*
> *--*
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
