[jira] [Updated] (HIVE-20972) Enable TestMiniLlapLocalCliDriver cbo_limit and cbo_rp_limit
[ https://issues.apache.org/jira/browse/HIVE-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20972: Attachment: HIVE-20972.03.patch > Enable TestMiniLlapLocalCliDriver cbo_limit and cbo_rp_limit > > > Key: HIVE-20972 > URL: https://issues.apache.org/jira/browse/HIVE-20972 > Project: Hive > Issue Type: Test >Reporter: Vihang Karajgaonkar >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20972.01.patch, HIVE-20972.02.patch, > HIVE-20972.02.patch, HIVE-20972.03.patch, HIVE-20972.03.patch, > HIVE-20972.03.patch, HIVE-20972.03.patch, HIVE-20972.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22624) Fix results_cache_invalidation and results_cache_lifetime
[
https://issues.apache.org/jira/browse/HIVE-22624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994347#comment-16994347
]
Zoltan Haindrich commented on HIVE-22624:
-
[~abstractdog] Could you please take a look?
> Fix results_cache_invalidation and results_cache_lifetime
> -
>
> Key: HIVE-22624
> URL: https://issues.apache.org/jira/browse/HIVE-22624
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22624.01.patch, logs.tgz
>
>
> these tests are falling together
> * they both run on hive-ptest-slaves-8e2.c.gcp-hive-upstream.internal
> * I suspect that they are fighting for the ownership /tmp/hive/_resultscache_
> exception#1
> {code}
> 2019-12-11T00:16:02,564 INFO [22d96536-983a-417d-b10c-aa1766ba0994 main]
> results.QueryResultsCache: Initializing query results cache at
> /tmp/hive/_resultscache_
> 2019-12-11T00:16:02,569 DEBUG [22d96536-983a-417d-b10c-aa1766ba0994 main]
> exec.Utilities: HDFS dir: /tmp/hive/_resultscache_ with schema null,
> permission: rwxr-xr-x
> 2019-12-11T00:16:02,571 ERROR [22d96536-983a-417d-b10c-aa1766ba0994 main]
> ql.Driver: FAILED: IllegalStateException java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Curren
> t permissions are: rwxr-xr-x
> java.lang.IllegalStateException: java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Current permissions are:
> rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.checkResultsCache(SemanticAnalyzer.java:15230)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12467)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:359)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> {code}
> exception#2
> {code}
> 2019-12-11T00:15:06,156 ERROR [5f435468-f02b-4587-ae48-51bfdbed main]
> ql.Driver: FAILED: IllegalStateException java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Curren
> t permissions are: rwxr-xr-x
> java.lang.IllegalStateException: java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Current permissions are:
> rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.checkResultsCache(SemanticAnalyzer.java:15230)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12467)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:359)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22510) Support decimal64 operations for column operands with different scales
[
https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994342#comment-16994342
]
Hive QA commented on HIVE-22510:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} ql: The patch generated 1 new + 780 unchanged - 0
fixed = 781 total (was 780) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19887/dev-support/hive-personality.sh
|
| git revision | master / 4a039ad |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19887/yetus/diff-checkstyle-ql.txt
|
| modules | C: vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19887/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support decimal64 operations for column operands with different scales
> --
>
> Key: HIVE-22510
> URL: https://issues.apache.org/jira/browse/HIVE-22510
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22510.11.patch, HIVE-22510.13.patch,
> HIVE-22510.2.patch, HIVE-22510.3.patch, HIVE-22510.4.patch,
> HIVE-22510.5.patch, HIVE-22510.7.patch, HIVE-22510.9.patch
>
>
> Right now, if the operands on the decimal64 operations are columns with
> different scales, then we do not use the decimal64 vectorized version and
> fall back to HiveDecimal vectorized version of the operator. In this Jira, we
> will check if we can use decimal64 vectorized version, even if the scales are
> different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22632) Improve estimateRowSizeFromSchema
[ https://issues.apache.org/jira/browse/HIVE-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22632: --- Status: Patch Available (was: Open) > Improve estimateRowSizeFromSchema > - > > Key: HIVE-22632 > URL: https://issues.apache.org/jira/browse/HIVE-22632 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22632.1.patch > > > estimateRowSizeFromSchema un-necessarily iterate and do look-up. This could > be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22632) Improve estimateRowSizeFromSchema
[ https://issues.apache.org/jira/browse/HIVE-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22632: --- Attachment: HIVE-22632.1.patch > Improve estimateRowSizeFromSchema > - > > Key: HIVE-22632 > URL: https://issues.apache.org/jira/browse/HIVE-22632 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22632.1.patch > > > estimateRowSizeFromSchema un-necessarily iterate and do look-up. This could > be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22632) Improve estimateRowSizeFromSchema
[ https://issues.apache.org/jira/browse/HIVE-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-22632: -- > Improve estimateRowSizeFromSchema > - > > Key: HIVE-22632 > URL: https://issues.apache.org/jira/browse/HIVE-22632 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > estimateRowSizeFromSchema un-necessarily iterate and do look-up. This could > be avoided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21917) COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs
[ https://issues.apache.org/jira/browse/HIVE-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994322#comment-16994322 ] Aditya Shah commented on HIVE-21917: [~pvary] the follow-up fix is HIVE-22625. Can you please take a look at that too. Thanks, Aditya > COMPLETED_TXN_COMPONENTS table is never cleaned up unless Compactor runs > > > Key: HIVE-21917 > URL: https://issues.apache.org/jira/browse/HIVE-21917 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0, 3.1.1 >Reporter: Craig Condit >Assignee: Denys Kuzmenko >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21917.1.patch, HIVE-21917.2.patch, > HIVE-21917.3.patch, HIVE-21917.4.patch, HIVE-21917.5.patch, HIVE-21917.6.patch > > > The Initiator thread in the metastore repeatedly loops over entries in the > COMPLETED_TXN_COMPONENTS table to determine which partitions / tables might > need to be compacted. However, entries are never removed from this table > except by a completed Compactor run. > In a cluster where most tables / partitions are write-once read-many, this > results in stale entries in this table never being cleaned up. In a small > test cluster, we have observed approximately 45k entries in this table > (virtually equal to the number of partitions in the cluster) while < 100 of > these tables have delta files at all. Since most of the tables will never get > enough writes to trigger a compaction (and in fact have only ever been > written to once), the initiator thread keeps trying to evaluate them on every > loop. > On this test cluster, it takes approximately 10 minutes to loop through all > the entries and results in severe performance degradation on metastore > operations. With the default run timing of 5 minutes, the initiator basically > never stops running. > On a production cluster with 2M partitions, this would be a non-starter. > The initiator thread should proactively remove entries from > COMPLETED_TXN_COMPONENTS when it determines that a compaction is not needed, > so that they are not evaluated again on the next loop. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22631) Avoid deep copying partition list in listPartitionsByExpr
[ https://issues.apache.org/jira/browse/HIVE-22631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22631: --- Attachment: HIVE-22631.1.patch > Avoid deep copying partition list in listPartitionsByExpr > - > > Key: HIVE-22631 > URL: https://issues.apache.org/jira/browse/HIVE-22631 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22631.1.patch > > > This is an expensive call, I am not sure why deepCopy is required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22631) Avoid deep copying partition list in listPartitionsByExpr
[ https://issues.apache.org/jira/browse/HIVE-22631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22631: --- Status: Patch Available (was: Open) > Avoid deep copying partition list in listPartitionsByExpr > - > > Key: HIVE-22631 > URL: https://issues.apache.org/jira/browse/HIVE-22631 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22631.1.patch > > > This is an expensive call, I am not sure why deepCopy is required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22631) Avoid deep copying partition list in listPartitionsByExpr
[ https://issues.apache.org/jira/browse/HIVE-22631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-22631: -- > Avoid deep copying partition list in listPartitionsByExpr > - > > Key: HIVE-22631 > URL: https://issues.apache.org/jira/browse/HIVE-22631 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > This is an expensive call, I am not sure why deepCopy is required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994300#comment-16994300
]
Hive QA commented on HIVE-22629:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988630/HIVE-22629.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19886/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19886/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19886/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12988630/HIVE-22629.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988630 - PreCommit-HIVE-Build
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch,
> HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994299#comment-16994299
]
Hive QA commented on HIVE-22629:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988630/HIVE-22629.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17770 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=225)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19885/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19885/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19885/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988630 - PreCommit-HIVE-Build
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch,
> HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22594: --- Assignee: John Sherman (was: Zoltan Haindrich) > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch, > HIVE-22594.6.patch, HIVE-22594.6.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22594: Attachment: HIVE-22594.6.patch > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch, > HIVE-22594.6.patch, HIVE-22594.6.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22594: --- Assignee: Zoltan Haindrich (was: John Sherman) > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch, > HIVE-22594.6.patch, HIVE-22594.6.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994266#comment-16994266
]
Hive QA commented on HIVE-22629:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 376 unchanged - 1
fixed = 376 total (was 377) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19885/dev-support/hive-personality.sh
|
| git revision | master / 4a039ad |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19885/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch,
> HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22584:
--
Attachment: HIVE-22584.3.patch
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.2.patch, HIVE-22584.3.patch, HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22617) Re-Enable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[
https://issues.apache.org/jira/browse/HIVE-22617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994236#comment-16994236
]
Hive QA commented on HIVE-22617:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988546/HIVE-22617.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17751 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_wrong_db]
(batchId=103)
org.apache.hive.service.server.TestHS2HttpServerPam.org.apache.hive.service.server.TestHS2HttpServerPam
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19884/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19884/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19884/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988546 - PreCommit-HIVE-Build
> Re-Enable PreCommit test
> org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
> ---
>
> Key: HIVE-22617
> URL: https://issues.apache.org/jira/browse/HIVE-22617
> Project: Hive
> Issue Type: Sub-task
>Reporter: Oliver Draese
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22617.01.patch, hive.log, initialize.log
>
>
> The test was disabled via HIVE-22616 because it was flaky. If the test is
> considered valid, it needs to be fixed and re-enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22594: -- Assignee: John Sherman (was: Jesus Camacho Rodriguez) > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch, HIVE-22594.6.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22594: --- Attachment: HIVE-22594.6.patch > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch, HIVE-22594.6.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22594: -- Assignee: Jesus Camacho Rodriguez (was: John Sherman) > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch, HIVE-22594.6.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22572: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master since test was failing due to missing regenerated q file in another patch. Thanks [~odraese] > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch, > HIVE-22572.6.patch, HIVE-22572.7.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?focusedWorklogId=358290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358290 ] ASF GitHub Bot logged work on HIVE-22408: - Author: ASF GitHub Bot Created on: 12/Dec/19 04:51 Start Date: 12/Dec/19 04:51 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #860: HIVE-22408 The fix for CALCITE-2991 creates wrong results on edge case URL: https://github.com/apache/hive/pull/860#discussion_r356957397 ## File path: ql/src/test/results/clientpositive/llap/mapjoin_hint.q.out ## @@ -648,7 +651,7 @@ STAGE PLANS: outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 input vertices: 1 Reducer 3 -Statistics: Num rows: 1 Data size: 816 Basic stats: COMPLETE Column stats: NONE +Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Review comment: @kgyrtkirk I guess Basic stats:NONE is not really correct since we do have full stats here. But since number of rows is zero I don't think there should be any side effects. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358290) Time Spent: 50m (was: 40m) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[
https://issues.apache.org/jira/browse/HIVE-22408?focusedWorklogId=358289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358289
]
ASF GitHub Bot logged work on HIVE-22408:
-
Author: ASF GitHub Bot
Created on: 12/Dec/19 04:49
Start Date: 12/Dec/19 04:49
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #860: HIVE-22408
The fix for CALCITE-2991 creates wrong results on edge case
URL: https://github.com/apache/hive/pull/860#discussion_r356957099
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitRemoveRule.java
##
@@ -41,8 +42,17 @@ public boolean matches(RelOptRuleCall call) {
final HiveSortLimit sortLimit = call.rel(0);
Double maxRowCount =
call.getMetadataQuery().getMaxRowCount(sortLimit.getInput());
-if (maxRowCount != null &&(maxRowCount <= 1)) {
- return true;
+if (maxRowCount != null) {
+ if (sortLimit.getFetchExpr() != null) {
+// we have LIMIT
+int limit = RexLiteral.intValue(sortLimit.getFetchExpr());
+if (maxRowCount <= limit) {
Review comment:
@kgyrtkirk `MaxRowCount` is guaranteed to return the number of rows which
will be >= the actual number of rows, therefore we should never end up removing
LIMIT when we shoudn't
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 358289)
Time Spent: 40m (was: 0.5h)
> The fix for CALCITE-2991 creates wrong results on edge case
> ---
>
> Key: HIVE-22408
> URL: https://issues.apache.org/jira/browse/HIVE-22408
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch,
> HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This
> patch causes a failure in subquery_ALL.q
> In order to proceed with the upgrade, the patch was bypassed by creating the
> HiveRelMdMaxRowCount class which overrides the behavior in Calcite.
> In order to recreate the regression, remove this file and the references to
> the class in HiveDefaultRelMetadataProvider.
> This Jira is being created before the HIVE-22274 PR has been pushed, so
> please wait until that is pushed before working on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22597) Include More Context in Database NoSuchObjectException
[
https://issues.apache.org/jira/browse/HIVE-22597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22597:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Include More Context in Database NoSuchObjectException
> --
>
> Key: HIVE-22597
> URL: https://issues.apache.org/jira/browse/HIVE-22597
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22597.1.patch, HIVE-22597.2.patch
>
>
> {code:none}
> org.apache.hadoop.hive.metastore.api.NoSuchObjectException: default
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:717)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor260.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_102]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-standalone-metastore-server-4.0.0-SN
> {code}
> One can decipher that this exception is in regards to a database by looking
> at the stack trace, but it should be specified in the error message itself.
> Also, there is no catalogue information provided, so it could be a bit
> ambiguous.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994183#comment-16994183 ] Vineet Garg commented on HIVE-22629: Looks good to me. +1 pending tests. > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch, > HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22617) Re-Enable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[
https://issues.apache.org/jira/browse/HIVE-22617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994180#comment-16994180
]
Hive QA commented on HIVE-22617:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
53s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19884/dev-support/hive-personality.sh
|
| git revision | master / df63e3a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: itests/util itests/hive-unit U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19884/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Re-Enable PreCommit test
> org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
> ---
>
> Key: HIVE-22617
> URL: https://issues.apache.org/jira/browse/HIVE-22617
> Project: Hive
> Issue Type: Sub-task
>Reporter: Oliver Draese
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-22617.01.patch, hive.log, initialize.log
>
>
> The test was disabled via HIVE-22616 because it was flaky. If the test is
> considered valid, it needs to be fixed and re-enabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22508) KeyWrapperComparator throws exception
[ https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22508: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~kkasa]! > KeyWrapperComparator throws exception > - > > Key: HIVE-22508 > URL: https://issues.apache.org/jira/browse/HIVE-22508 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch, > HIVE-22508.1.patch, HIVE-22508.2.patch, HIVE-22508.2.patch, > HIVE-22508.2.patch, HIVE-22508.2.patch > > > TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a > copied key should be compared. > The current implementation uses the standard object inspectors for all > KeyWrapper instances. However when comparing untouched KeyWrappers the key > object inspector should be used which can be extracted form > Operator.inputObjectInspectors during initialization of the key's > ExprNodeEvaluator. > This can cause a ClassCastException when the comparator is used collections > like TreeSet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22594: Attachment: HIVE-22594.5.patch > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch, HIVE-22594.5.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22589) Add storage support for ProlepticCalendar
[
https://issues.apache.org/jira/browse/HIVE-22589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22589:
---
Attachment: HIVE-22589.02.patch
> Add storage support for ProlepticCalendar
> -
>
> Key: HIVE-22589
> URL: https://issues.apache.org/jira/browse/HIVE-22589
> Project: Hive
> Issue Type: Bug
> Components: Avro, ORC, Parquet
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0, 3.2.0, 3.1.3
>
> Attachments: HIVE-22589.01.patch, HIVE-22589.02.patch,
> HIVE-22589.patch, HIVE-22589.patch
>
>
> Hive recently moved its processing to the proleptic calendar, which has
> created some issues for users who have dates before 1580 AD.
> I'd propose extending the column vectors for times & dates to encode which
> calendar they are using.
> * create DateColumnVector that extends LongColumnVector
> * add a method to change calendars to both DateColumnVector and
> TimestampColumnVector.
> {code}
> /**
>* Change the calendar to or from proleptic. If the new and old values of
> the flag are the
>* same, nothing is done.
>* useProleptic - set the flag for the proleptic calendar
>* updateData - change the data to match the new value of the flag.
>*/
> void changeCalendar(useProleptic: boolean, updateData: boolean);
> /**
>* Detect whether this data is using the proleptic calendar.
>*/
> boolean usingProlepticCalendar();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22594) Reorder metastore thrift structures
[
https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994160#comment-16994160
]
Hive QA commented on HIVE-22594:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988562/HIVE-22594.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17753 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_wrong_db]
(batchId=103)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19883/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19883/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19883/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988562 - PreCommit-HIVE-Build
> Reorder metastore thrift structures
> ---
>
> Key: HIVE-22594
> URL: https://issues.apache.org/jira/browse/HIVE-22594
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Thrift API
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch,
> HIVE-22594.3.patch, HIVE-22594.4.patch
>
>
> Currently hive_metastore.thrift C++ output does not compile due to structures
> like ScheduledQueryKey and ColumnStatistics being referenced before being
> defined later in the file. We should reorder the structures in the file so
> the C++ output is usable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22609) Reduce number of FS getFileStatus calls in AcidUtils::getHdfsDirSnapshots
[
https://issues.apache.org/jira/browse/HIVE-22609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22609:
Attachment: HIVE-22609.2.patch
> Reduce number of FS getFileStatus calls in AcidUtils::getHdfsDirSnapshots
> -
>
> Key: HIVE-22609
> URL: https://issues.apache.org/jira/browse/HIVE-22609
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22609.1.patch, HIVE-22609.2.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L1380]
> ACID delta folder contains {{_orc_acid_version}} and {{bucket_0}} files.
> For both these files, parent dir is the same. Number of getFileStatus in such
> cases should be reduced by 1/2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22548) Optimise Utilities.removeTempOrDuplicateFiles when moving files to final location
[
https://issues.apache.org/jira/browse/HIVE-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994129#comment-16994129
]
Rajesh Balamohan commented on HIVE-22548:
-
LGTM. +1.
Please add the {{LOG.debug}} statement before commit.
> Optimise Utilities.removeTempOrDuplicateFiles when moving files to final
> location
> -
>
> Key: HIVE-22548
> URL: https://issues.apache.org/jira/browse/HIVE-22548
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Attachments: HIVE-22548.01.patch, HIVE-22548.02.patch
>
>
> {{Utilities.removeTempOrDuplicateFiles}}
> is very slow with cloud storage, as it executes {{listStatus}} twice and also
> runs in single threaded mode.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1629
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22594) Reorder metastore thrift structures
[
https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994123#comment-16994123
]
Hive QA commented on HIVE-22594:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19883/dev-support/hive-personality.sh
|
| git revision | master / df63e3a |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19883/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reorder metastore thrift structures
> ---
>
> Key: HIVE-22594
> URL: https://issues.apache.org/jira/browse/HIVE-22594
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Thrift API
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch,
> HIVE-22594.3.patch, HIVE-22594.4.patch
>
>
> Currently hive_metastore.thrift C++ output does not compile due to structures
> like ScheduledQueryKey and ColumnStatistics being referenced before being
> defined later in the file. We should reorder the structures in the file so
> the C++ output is usable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22609) Reduce number of FS getFileStatus calls in AcidUtils::getHdfsDirSnapshots
[
https://issues.apache.org/jira/browse/HIVE-22609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22609:
Status: Patch Available (was: Open)
> Reduce number of FS getFileStatus calls in AcidUtils::getHdfsDirSnapshots
> -
>
> Key: HIVE-22609
> URL: https://issues.apache.org/jira/browse/HIVE-22609
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22609.1.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L1380]
> ACID delta folder contains {{_orc_acid_version}} and {{bucket_0}} files.
> For both these files, parent dir is the same. Number of getFileStatus in such
> cases should be reduced by 1/2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22609) Reduce number of FS getFileStatus calls in AcidUtils::getHdfsDirSnapshots
[
https://issues.apache.org/jira/browse/HIVE-22609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22609:
Attachment: HIVE-22609.1.patch
> Reduce number of FS getFileStatus calls in AcidUtils::getHdfsDirSnapshots
> -
>
> Key: HIVE-22609
> URL: https://issues.apache.org/jira/browse/HIVE-22609
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-22609.1.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L1380]
> ACID delta folder contains {{_orc_acid_version}} and {{bucket_0}} files.
> For both these files, parent dir is the same. Number of getFileStatus in such
> cases should be reduced by 1/2.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22629: -- Attachment: HIVE-22629.2.patch > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch, > HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994111#comment-16994111 ] Slim Bouguerra commented on HIVE-22629: --- [~belugabehr] thanks sounds good to me. > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.1.patch, HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994109#comment-16994109
]
Hive QA commented on HIVE-22572:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988560/HIVE-22572.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17753 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_wrong_db]
(batchId=103)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19882/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19882/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19882/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988560 - PreCommit-HIVE-Build
> NullPointerException when using dynamic semijoin reduction
> --
>
> Key: HIVE-22572
> URL: https://issues.apache.org/jira/browse/HIVE-22572
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch,
> HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch,
> HIVE-22572.6.patch, HIVE-22572.7.patch, HIVE-22572.patch
>
>
> From HS2 logs
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22497) Remove default value for Capabilities from HiveConf
[ https://issues.apache.org/jira/browse/HIVE-22497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22497: - Resolution: Fixed Status: Resolved (was: Patch Available) Fix has been committed to master. Thank you for the commit [~maheshk114] and the review [~thejas] and [~samuelan]. Closing the jira. > Remove default value for Capabilities from HiveConf > --- > > Key: HIVE-22497 > URL: https://issues.apache.org/jira/browse/HIVE-22497 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22497.patch > > > This class is used and bundled in other jars that 3rd party connectors like > teradata etc. So it would be good to remove this default value from HiveConf > but rely on it being set in hive-site.xml instead. The HiveServer2 should > still set this as part of HS2 initialization or via hiveserver2-site.xml -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994064#comment-16994064
]
Hive QA commented on HIVE-22572:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19882/dev-support/hive-personality.sh
|
| git revision | master / e959d4a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19882/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19882/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NullPointerException when using dynamic semijoin reduction
> --
>
> Key: HIVE-22572
> URL: https://issues.apache.org/jira/browse/HIVE-22572
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch,
> HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch,
> HIVE-22572.6.patch, HIVE-22572.7.patch, HIVE-22572.patch
>
>
> From HS2 logs
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487)
> ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1]
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360)
>
[jira] [Commented] (HIVE-17680) TestNegativeCliDriver[merge_negative_5.q]
[
https://issues.apache.org/jira/browse/HIVE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994037#comment-16994037
]
Hive QA commented on HIVE-17680:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988554/HIVE-17680.01.branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10727 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=117)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19881/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19881/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19881/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988554 - PreCommit-HIVE-Build
> TestNegativeCliDriver[merge_negative_5.q]
> -
>
> Key: HIVE-17680
> URL: https://issues.apache.org/jira/browse/HIVE-17680
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-17680.01.branch-2.patch
>
>
> Seeing this error stack in the hive.log when I run it locally
> Test failed in
> https://builds.apache.org/job/PreCommit-HIVE-Build/7083/testReport
> {noformat}
> 2017-10-03T10:34:59,335 ERROR [d6419a0a-1fc9-4454-9575-f085c7f2cb55 main]
> metastore.RetryingHMSHandler: MetaException(message:The table must be
> bucketed and stored using an ACID compliant format (such as ORC))
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:199)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:68)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:53)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2290)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1413)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1511)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> com.sun.proxy.$Proxy34.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2407)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:760)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:748)
> at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Commented] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994019#comment-16994019
]
Hive QA commented on HIVE-22584:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988555/HIVE-22584.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17742 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniLlapArrow - did not produce a TEST-*.xml file (likely timed
out) (batchId=286)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_wrong_db]
(batchId=103)
org.apache.hive.jdbc.TestRestrictedList.testRestrictedList (batchId=286)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19880/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19880/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19880/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988555 - PreCommit-HIVE-Build
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.2.patch, HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22630) Do not retrieve Materialized View definitions for rebuild if query is test SQL
[ https://issues.apache.org/jira/browse/HIVE-22630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22630: -- Summary: Do not retrieve Materialized View definitions for rebuild if query is test SQL (was: Do not retrieve Materialized View definition for rebuild if query is test SQL) > Do not retrieve Materialized View definitions for rebuild if query is test SQL > -- > > Key: HIVE-22630 > URL: https://issues.apache.org/jira/browse/HIVE-22630 > Project: Hive > Issue Type: Bug > Environment: Hive-3.1.2 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22630.patch > > > for the query like select 1, select current_timestamp, select current_date > hive retrieve all the Materialized view from megastore, if the no of > databases are too large then this call take lots of time, the situation > becomes worse if there are too frequent if hive server receives frequent > "select 1" query ( connection pool uses it to check if the connection is > valid or not). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22630) Do not retrieve Materialized View definition for rebuild if query is test SQL
[ https://issues.apache.org/jira/browse/HIVE-22630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22630: -- Attachment: HIVE-22630.patch Status: Patch Available (was: Open) > Do not retrieve Materialized View definition for rebuild if query is test SQL > - > > Key: HIVE-22630 > URL: https://issues.apache.org/jira/browse/HIVE-22630 > Project: Hive > Issue Type: Bug > Environment: Hive-3.1.2 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22630.patch > > > for the query like select 1, select current_timestamp, select current_date > hive retrieve all the Materialized view from megastore, if the no of > databases are too large then this call take lots of time, the situation > becomes worse if there are too frequent if hive server receives frequent > "select 1" query ( connection pool uses it to check if the connection is > valid or not). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22630) Do not retrieve Materialized View definition for rebuild if query is test SQL
[ https://issues.apache.org/jira/browse/HIVE-22630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh reassigned HIVE-22630: - > Do not retrieve Materialized View definition for rebuild if query is test SQL > - > > Key: HIVE-22630 > URL: https://issues.apache.org/jira/browse/HIVE-22630 > Project: Hive > Issue Type: Bug > Environment: Hive-3.1.2 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > > for the query like select 1, select current_timestamp, select current_date > hive retrieve all the Materialized view from megastore, if the no of > databases are too large then this call take lots of time, the situation > becomes worse if there are too frequent if hive server receives frequent > "select 1" query ( connection pool uses it to check if the connection is > valid or not). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993991#comment-16993991
]
Hive QA commented on HIVE-22584:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
49s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19880/dev-support/hive-personality.sh
|
| git revision | master / e959d4a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19880/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19880/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.2.patch, HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
>
[jira] [Commented] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993982#comment-16993982 ] Jesus Camacho Rodriguez commented on HIVE-22594: +1 (pending tests) > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22603) TestSparkCliDriver[vectorization_input_format_excludes.q]
[
https://issues.apache.org/jira/browse/HIVE-22603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993969#comment-16993969
]
Hive QA commented on HIVE-22603:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988551/HIVE-22603.01.branch-2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=227)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_tableproperty_optimize]
(batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=155)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19879/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19879/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19879/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988551 - PreCommit-HIVE-Build
> TestSparkCliDriver[vectorization_input_format_excludes.q]
> -
>
> Key: HIVE-22603
> URL: https://issues.apache.org/jira/browse/HIVE-22603
> Project: Hive
> Issue Type: Sub-task
>Reporter: László Bodor
>Priority: Major
> Attachments: HIVE-22603.01.branch-2.patch,
> vectorization_input_format_excludes.q.out.diff
>
>
> attached q.out diff: [^vectorization_input_format_excludes.q.out.diff]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22593) Dynamically partitioned MM (insert-only ACID) tables don't compact automatically
[
https://issues.apache.org/jira/browse/HIVE-22593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993937#comment-16993937
]
Hive QA commented on HIVE-22593:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988547/HIVE-22593.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17754 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=75)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication
(batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19878/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19878/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19878/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988547 - PreCommit-HIVE-Build
> Dynamically partitioned MM (insert-only ACID) tables don't compact
> automatically
>
>
> Key: HIVE-22593
> URL: https://issues.apache.org/jira/browse/HIVE-22593
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22593.01.patch, HIVE-22593.01.patch,
> HIVE-22593.02.patch
>
>
> Dynamic partitions of MM tables aren't entered into the HMS table
> TXN_COMPONENTS. On inserting into such tables we see this line in the HMS log:
> {code:java}
> Expected to move at least one record from txn_components to
> completed_txn_components when committing txn!{code}
> (This is not the case for non-partitioned MM tables.)
> Since the partitions aren't entered into COMPLETED_TXN_COMPONENTS, they
> aren't considered for automatic compaction.
> Probably the culprit is
> org.apache.hadoop.hive.ql.metadata.Hive#loadDynamicPartitions which has an
> isAcid parameter that is always false regarding MM tables, and also because
> MM tables' "write type" is AcidUtils.Operation.NOT_ACID and not INSERT.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22593) Dynamically partitioned MM (insert-only ACID) tables don't compact automatically
[
https://issues.apache.org/jira/browse/HIVE-22593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993908#comment-16993908
]
Hive QA commented on HIVE-22593:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19878/dev-support/hive-personality.sh
|
| git revision | master / eb72a0c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19878/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19878/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamically partitioned MM (insert-only ACID) tables don't compact
> automatically
>
>
> Key: HIVE-22593
> URL: https://issues.apache.org/jira/browse/HIVE-22593
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-22593.01.patch, HIVE-22593.01.patch,
> HIVE-22593.02.patch
>
>
> Dynamic partitions of MM tables aren't entered into the HMS table
> TXN_COMPONENTS. On inserting into such tables we see this line in the HMS log:
> {code:java}
> Expected to move at least one record from txn_components to
> completed_txn_components when committing txn!{code}
> (This is not the case for non-partitioned MM tables.)
> Since the partitions aren't entered into COMPLETED_TXN_COMPONENTS, they
> aren't considered for automatic compaction.
> Probably the culprit is
> org.apache.hadoop.hive.ql.metadata.Hive#loadDynamicPartitions which has an
> isAcid parameter that is always false regarding MM tables, and also because
> MM tables' "write type" is AcidUtils.Operation.NOT_ACID and not INSERT.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993895#comment-16993895
]
David Mollitor commented on HIVE-22629:
---
[~bslim] OK, but should still return...
{code:java}
return new ArrayList<>(super.getChildren());
{code}
Going to be much quicker and efficient than iterating the list and copying
one-by-one
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.1.patch, HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22624) Fix results_cache_invalidation and results_cache_lifetime
[
https://issues.apache.org/jira/browse/HIVE-22624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993889#comment-16993889
]
Hive QA commented on HIVE-22624:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988523/HIVE-22624.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17755 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19877/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19877/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19877/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988523 - PreCommit-HIVE-Build
> Fix results_cache_invalidation and results_cache_lifetime
> -
>
> Key: HIVE-22624
> URL: https://issues.apache.org/jira/browse/HIVE-22624
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22624.01.patch, logs.tgz
>
>
> these tests are falling together
> * they both run on hive-ptest-slaves-8e2.c.gcp-hive-upstream.internal
> * I suspect that they are fighting for the ownership /tmp/hive/_resultscache_
> exception#1
> {code}
> 2019-12-11T00:16:02,564 INFO [22d96536-983a-417d-b10c-aa1766ba0994 main]
> results.QueryResultsCache: Initializing query results cache at
> /tmp/hive/_resultscache_
> 2019-12-11T00:16:02,569 DEBUG [22d96536-983a-417d-b10c-aa1766ba0994 main]
> exec.Utilities: HDFS dir: /tmp/hive/_resultscache_ with schema null,
> permission: rwxr-xr-x
> 2019-12-11T00:16:02,571 ERROR [22d96536-983a-417d-b10c-aa1766ba0994 main]
> ql.Driver: FAILED: IllegalStateException java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Curren
> t permissions are: rwxr-xr-x
> java.lang.IllegalStateException: java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Current permissions are:
> rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.checkResultsCache(SemanticAnalyzer.java:15230)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12467)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:359)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:171)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> {code}
> exception#2
> {code}
> 2019-12-11T00:15:06,156 ERROR [5f435468-f02b-4587-ae48-51bfdbed main]
> ql.Driver: FAILED: IllegalStateException java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Curren
> t permissions are: rwxr-xr-x
> java.lang.IllegalStateException: java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Current permissions are:
> rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.checkResultsCache(SemanticAnalyzer.java:15230)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12467)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:359)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993886#comment-16993886 ] Slim Bouguerra commented on HIVE-22629: --- [~belugabehr] what you are saying make sense, but that is out of scope of this patch. You need to dive deep and see if this list is mutated later and what null is used for... > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.1.patch, HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22629: -- Attachment: HIVE-22629.1.patch > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.1.patch, HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993885#comment-16993885
]
Slim Bouguerra commented on HIVE-22629:
---
also this is happening at SemanticAnalyzer {code} processPositionAlias{code}
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993884#comment-16993884 ] Ramesh Kumar Thangarajan commented on HIVE-22510: - Hi [~gopalv], Can you please review the PR for the decimal64 operations on different scale? [https://github.com/apache/hive/pull/864] > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.13.patch, > HIVE-22510.2.patch, HIVE-22510.3.patch, HIVE-22510.4.patch, > HIVE-22510.5.patch, HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993857#comment-16993857 ] Anant Mittal commented on HIVE-18922: - Thanks David!! It is possible that these are the cause, I will confirm once I can get this tried on even newer Hive versions. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. > Update: It seems HIVE-11940 did not cover this as the issue is seen in later > versions too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22624) Fix results_cache_invalidation and results_cache_lifetime
[
https://issues.apache.org/jira/browse/HIVE-22624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993855#comment-16993855
]
Hive QA commented on HIVE-22624:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/util: The patch generated 1 new + 54 unchanged
- 1 fixed = 55 total (was 55) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19877/dev-support/hive-personality.sh
|
| git revision | master / eb72a0c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19877/yetus/diff-checkstyle-itests_util.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19877/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/util U: itests/util |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19877/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix results_cache_invalidation and results_cache_lifetime
> -
>
> Key: HIVE-22624
> URL: https://issues.apache.org/jira/browse/HIVE-22624
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22624.01.patch, logs.tgz
>
>
> these tests are falling together
> * they both run on hive-ptest-slaves-8e2.c.gcp-hive-upstream.internal
> * I suspect that they are fighting for the ownership /tmp/hive/_resultscache_
> exception#1
> {code}
> 2019-12-11T00:16:02,564 INFO [22d96536-983a-417d-b10c-aa1766ba0994 main]
> results.QueryResultsCache: Initializing query results cache at
> /tmp/hive/_resultscache_
> 2019-12-11T00:16:02,569 DEBUG [22d96536-983a-417d-b10c-aa1766ba0994 main]
> exec.Utilities: HDFS dir: /tmp/hive/_resultscache_ with schema null,
> permission: rwxr-xr-x
> 2019-12-11T00:16:02,571 ERROR [22d96536-983a-417d-b10c-aa1766ba0994 main]
> ql.Driver: FAILED: IllegalStateException java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Curren
> t permissions are: rwxr-xr-x
> java.lang.IllegalStateException: java.lang.RuntimeException: The dir:
> /tmp/hive/_resultscache_ on HDFS should be writable. Current permissions are:
> rwxr-xr-x
>
[jira] [Comment Edited] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993853#comment-16993853
]
David Mollitor edited comment on HIVE-22629 at 12/11/19 7:44 PM:
-
Hey.
Why not just...
{code:java}
/*
* (non-Javadoc)
*
* @see org.apache.hadoop.hive.ql.lib.Node#getChildren()
*/
@Override
public List getChildren() {
List children = super.getChildren();
return (children == null) ? Collections.emptyList() :
Collections.unmodifiableList(new ArrayList<>(super.getChildren()));
}
{code}
Returning null "sucks". If a node has no children, the list of children should
be empty. Otherwise it's ambiguous... what is an empty list v.s. null?
https://github.com/google/guava/wiki/UsingAndAvoidingNullExplained
was (Author: belugabehr):
Hey.
Why not just...
{code:java}
/*
* (non-Javadoc)
*
* @see org.apache.hadoop.hive.ql.lib.Node#getChildren()
*/
@Override
public ArrayList getChildren() {
if (super.getChildCount() == 0) {
return Collections.emptyList();
}
return Collections.unmodifiableList(new ArrayList<>(super.getChildren()));
}
{code}
Returning null "sucks". If a node has no children, the list of children should
be empty. Otherwise it's ambiguous... what is an empty list v.s. null?
https://github.com/google/guava/wiki/UsingAndAvoidingNullExplained
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993854#comment-16993854 ] David McGinnis edited comment on HIVE-18922 at 12/11/19 7:42 PM: - [~infinitymittal]: Could this be related to -HIVE-17479- or -HIVE-20409-? Both of these describe a similar issue. They were fixed in 3.0 and 4.0 respectively, and the two versions you mention only use Hive 2.X. was (Author: mcginnda): [~infinitymittal]: Could this be related to [HIVE-17479|https://issues.apache.org/jira/browse/HIVE-17479] or [HIVE-20409|https://issues.apache.org/jira/browse/HIVE-17479]? Both of these describe a similar issue. They were fixed in 3.0 and 4.0 respectively, and the two versions you mention only use Hive 2.X. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. > Update: It seems HIVE-11940 did not cover this as the issue is seen in later > versions too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993854#comment-16993854 ] David McGinnis commented on HIVE-18922: --- [~infinitymittal]: Could this be related to [HIVE-17479|https://issues.apache.org/jira/browse/HIVE-17479] or [HIVE-20409|https://issues.apache.org/jira/browse/HIVE-17479]? Both of these describe a similar issue. They were fixed in 3.0 and 4.0 respectively, and the two versions you mention only use Hive 2.X. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. > Update: It seems HIVE-11940 did not cover this as the issue is seen in later > versions too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993853#comment-16993853
]
David Mollitor commented on HIVE-22629:
---
Hey.
Why not just...
{code:java}
/*
* (non-Javadoc)
*
* @see org.apache.hadoop.hive.ql.lib.Node#getChildren()
*/
@Override
public ArrayList getChildren() {
if (super.getChildCount() == 0) {
return Collections.emptyList();
}
return Collections.unmodifiableList(new ArrayList<>(super.getChildren()));
}
{code}
Returning null "sucks". If a node has no children, the list of children should
be empty. Otherwise it's ambiguous... what is an empty list v.s. null?
https://github.com/google/guava/wiki/UsingAndAvoidingNullExplained
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.patch)
> Drop table involved in materialized view leaves the table in inconsistent
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
>
> If you try dropping a table which is part of the definition of a created
> materialized view, the table is not dropped, which is the desired state as it
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did
> not succeed, leaving it in an inconsistent state.
>
> Repro:
> ---
> 1) Create tables:
>
> {code:java}
> CREATE TABLE emps ( empid INT, deptno INT, name VARCHAR(256), salary
> FLOAT, hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES
> ('transactional'='true');
> CREATE TABLE depts ( deptno INT, deptname VARCHAR(256), locationid
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>
> 2) Create the VM:
>
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps
> JOIN depts ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>
> 3) Following is in backend database at this point:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | 81 | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>
> 4) Let's drop the 'emps' table:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO : Compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 0.05 seconds
> INFO : Executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b):
> drop table emps
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b);
> Time taken: 10.281 seconds
> INFO : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>
> 5) List tables:
>
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO : Compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.041 seconds
> INFO : Executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413):
> show tables
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413);
> Time taken: 0.016 seconds
> INFO : OK
> +---+
> | tab_name |
> +---+
> | depts |
> | emps |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>
> 6) Now, from the backend-db point of view:
>
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE |
> ++---+---+--+---+
> | 81 | 16 | NULL | emps | MANAGED_TABLE |
> | 83 | 16 | 83 | depts | MANAGED_TABLE |
> | 84 | 16 | 84 | mv1 | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>
> 7) From Metastore.log
>
> {code:java}
> 2019-12-02T20:00:25,545 INFO [pool-6-thread-195]: metastore.HiveMetaStore
> (HiveMetaStore.java:logInfo(907)) - 196:
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Attachment: HIVE-22510.13.patch Status: Patch Available (was: Open) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.13.patch, > HIVE-22510.2.patch, HIVE-22510.3.patch, HIVE-22510.4.patch, > HIVE-22510.5.patch, HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Status: Open (was: Patch Available) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.2.patch, > HIVE-22510.3.patch, HIVE-22510.4.patch, HIVE-22510.5.patch, > HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Attachment: (was: HIVE-22510.12.patch) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.2.patch, > HIVE-22510.3.patch, HIVE-22510.4.patch, HIVE-22510.5.patch, > HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Attachment: HIVE-22510.12.patch Status: Patch Available (was: Open) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.2.patch, > HIVE-22510.3.patch, HIVE-22510.4.patch, HIVE-22510.5.patch, > HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Attachment: (was: HIVE-22510.12.patch) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.2.patch, > HIVE-22510.3.patch, HIVE-22510.4.patch, HIVE-22510.5.patch, > HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22486) Send only accessed columns for masking policies request
[ https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22486: --- Attachment: HIVE-22486.90.patch > Send only accessed columns for masking policies request > --- > > Key: HIVE-22486 > URL: https://issues.apache.org/jira/browse/HIVE-22486 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22486.01.patch, HIVE-22486.02.patch, > HIVE-22486.03.patch, HIVE-22486.05.patch, HIVE-22486.06.patch, > HIVE-22486.90.patch, HIVE-22486.90.patch, HIVE-22486.patch, > HIVE-22486.patchaddendum > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, we send all columns for masking request, even if they are not > accessed by the given query. We could send only those columns for which the > masking policy will be necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22629: -- Attachment: HIVE-22629.patch > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: HIVE-22629.patch, > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22486) Send only accessed columns for masking policies request
[
https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993838#comment-16993838
]
Hive QA commented on HIVE-22486:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988516/HIVE-22486.90.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17753 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[url_hook]
(batchId=296)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19876/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19876/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19876/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988516 - PreCommit-HIVE-Build
> Send only accessed columns for masking policies request
> ---
>
> Key: HIVE-22486
> URL: https://issues.apache.org/jira/browse/HIVE-22486
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22486.01.patch, HIVE-22486.02.patch,
> HIVE-22486.03.patch, HIVE-22486.05.patch, HIVE-22486.06.patch,
> HIVE-22486.90.patch, HIVE-22486.patch, HIVE-22486.patchaddendum
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently, we send all columns for masking request, even if they are not
> accessed by the given query. We could send only those columns for which the
> masking policy will be necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22629: -- Status: Patch Available (was: Open) > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra updated HIVE-22629: -- Attachment: noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > Attachments: > noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg > > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[ https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra reassigned HIVE-22629: - > AST Node Children can be quite expensive to build due to List resizing > -- > > Key: HIVE-22629 > URL: https://issues.apache.org/jira/browse/HIVE-22629 > Project: Hive > Issue Type: Improvement >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > > As per the attached profile, The AST Node can be a major source of CPU and > memory churn, due to the ArrayList resizing and copy. > In my Opinion this can be amortized by providing the actual size. > [~jcamachorodriguez] / [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22486) Send only accessed columns for masking policies request
[ https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993826#comment-16993826 ] Jesus Camacho Rodriguez commented on HIVE-22486: Thanks [~kgyrtkirk]! > Send only accessed columns for masking policies request > --- > > Key: HIVE-22486 > URL: https://issues.apache.org/jira/browse/HIVE-22486 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22486.01.patch, HIVE-22486.02.patch, > HIVE-22486.03.patch, HIVE-22486.05.patch, HIVE-22486.06.patch, > HIVE-22486.90.patch, HIVE-22486.patch, HIVE-22486.patchaddendum > > Time Spent: 1h > Remaining Estimate: 0h > > Currently, we send all columns for masking request, even if they are not > accessed by the given query. We could send only those columns for which the > masking policy will be necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22486) Send only accessed columns for masking policies request
[
https://issues.apache.org/jira/browse/HIVE-22486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993802#comment-16993802
]
Hive QA commented on HIVE-22486:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
38s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19876/dev-support/hive-personality.sh
|
| git revision | master / eb72a0c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19876/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19876/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Send only accessed columns for masking policies request
> ---
>
> Key: HIVE-22486
> URL: https://issues.apache.org/jira/browse/HIVE-22486
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Affects Versions: 4.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22486.01.patch, HIVE-22486.02.patch,
> HIVE-22486.03.patch, HIVE-22486.05.patch, HIVE-22486.06.patch,
> HIVE-22486.90.patch, HIVE-22486.patch, HIVE-22486.patchaddendum
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently, we send all columns for masking request, even if they are not
> accessed by the given query. We could send only those columns for which the
> masking policy will be necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Status: Open (was: Patch Available) > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.12.patch, > HIVE-22510.2.patch, HIVE-22510.3.patch, HIVE-22510.4.patch, > HIVE-22510.5.patch, HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22510) Support decimal64 operations for column operands with different scales
[ https://issues.apache.org/jira/browse/HIVE-22510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22510: Attachment: HIVE-22510.12.patch > Support decimal64 operations for column operands with different scales > -- > > Key: HIVE-22510 > URL: https://issues.apache.org/jira/browse/HIVE-22510 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22510.11.patch, HIVE-22510.12.patch, > HIVE-22510.2.patch, HIVE-22510.3.patch, HIVE-22510.4.patch, > HIVE-22510.5.patch, HIVE-22510.7.patch, HIVE-22510.9.patch > > > Right now, if the operands on the decimal64 operations are columns with > different scales, then we do not use the decimal64 vectorized version and > fall back to HiveDecimal vectorized version of the operator. In this Jira, we > will check if we can use decimal64 vectorized version, even if the scales are > different. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22601) Some columns will be lost when a UDTF has multiple aliases in some cases
[
https://issues.apache.org/jira/browse/HIVE-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993783#comment-16993783
]
okumin commented on HIVE-22601:
---
There is a bug in the step to rewrite TOK_SETCOLREFs in an AST.
A TOK_SETCOLREF is a placeholder to express column names.
In my case, the schema of the result of UNION ALL is unknown at the AST level.
So this placeholder will be required.
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L81]
This step assumes that there is only one alias per expression at most.
As a result, only c3 column was retained in my case.
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java#L450]
This is the syntax part and it will appear with a UDTF.
*
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/SelectClauseParser.g#L88]
*
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L4530-L4537]
In other cases where there are multiple children, a TOK_SELEXPR would have
tokens other than Identifier.
So my patch wouldn't break any case when there are these ones.
*
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/SelectClauseParser.g#L56-L58]
*
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/SelectClauseParser.g#L84]
*
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g#L155-L161]
*
[https://github.com/apache/hive/blob/eb72a0c01d3dc5dc50c220a548c7986a440eef3f/ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g#L334-L338]
Checkstyle errors came from existing incorrect format. I will revise them in
another ticket.
https://issues.apache.org/jira/browse/HIVE-22618
Concerning asflicense, it's not related to my patch.
> Some columns will be lost when a UDTF has multiple aliases in some cases
>
>
> Key: HIVE-22601
> URL: https://issues.apache.org/jira/browse/HIVE-22601
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.1.2
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22601.1.patch, HIVE-22601.2.patch, HIVE-22601.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Only one column will be retained when putting UDTFs with multiple aliases and
> a top-level UNION together.
> For example, the result of the following SQL should have three columns, c1,
> c2 and c3.
> {code:java}
> SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
> UNION ALL
> SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);
> {code}
> However, It's only the c3 column which I can get.
> {code:java}
> +-+
> | _u1.c3 |
> +-+
> | c |
> | f |
> +-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993768#comment-16993768
]
Hive QA commented on HIVE-21164:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988510/HIVE-21164.14.patch
{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17753 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19875/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19875/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19875/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988510 - PreCommit-HIVE-Build
> ACID: explore how we can avoid a move step during inserts/compaction
>
>
> Key: HIVE-21164
> URL: https://issues.apache.org/jira/browse/HIVE-21164
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch,
> HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch,
> HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch,
> HIVE-21164.2.patch, HIVE-21164.3.patch, HIVE-21164.4.patch,
> HIVE-21164.5.patch, HIVE-21164.6.patch, HIVE-21164.7.patch,
> HIVE-21164.8.patch, HIVE-21164.9.patch
>
>
> Currently, we write compacted data to a temporary location and then move the
> files to a final location, which is an expensive operation on some cloud file
> systems. Since HIVE-20823 is already in, it can control the visibility of
> compacted data for the readers. Therefore, we can perhaps avoid writing data
> to a temporary location and directly write compacted data to the intended
> final path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22601) Some columns will be lost when a UDTF has multiple aliases in some cases
[
https://issues.apache.org/jira/browse/HIVE-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-22601:
--
Labels: pull-request-available (was: )
> Some columns will be lost when a UDTF has multiple aliases in some cases
>
>
> Key: HIVE-22601
> URL: https://issues.apache.org/jira/browse/HIVE-22601
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.1.2
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22601.1.patch, HIVE-22601.2.patch, HIVE-22601.patch
>
>
> Only one column will be retained when putting UDTFs with multiple aliases and
> a top-level UNION together.
> For example, the result of the following SQL should have three columns, c1,
> c2 and c3.
> {code:java}
> SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
> UNION ALL
> SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);
> {code}
> However, It's only the c3 column which I can get.
> {code:java}
> +-+
> | _u1.c3 |
> +-+
> | c |
> | f |
> +-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22601) Some columns will be lost when a UDTF has multiple aliases in some cases
[
https://issues.apache.org/jira/browse/HIVE-22601?focusedWorklogId=357967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357967
]
ASF GitHub Bot logged work on HIVE-22601:
-
Author: ASF GitHub Bot
Created on: 11/Dec/19 17:33
Start Date: 11/Dec/19 17:33
Worklog Time Spent: 10m
Work Description: okumin commented on pull request #863: HIVE-22601 Some
columns will be lost when a UDTF has multiple aliases in some cases
URL: https://github.com/apache/hive/pull/863
https://issues.apache.org/jira/browse/HIVE-22601
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 357967)
Remaining Estimate: 0h
Time Spent: 10m
> Some columns will be lost when a UDTF has multiple aliases in some cases
>
>
> Key: HIVE-22601
> URL: https://issues.apache.org/jira/browse/HIVE-22601
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.1.2
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22601.1.patch, HIVE-22601.2.patch, HIVE-22601.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Only one column will be retained when putting UDTFs with multiple aliases and
> a top-level UNION together.
> For example, the result of the following SQL should have three columns, c1,
> c2 and c3.
> {code:java}
> SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
> UNION ALL
> SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);
> {code}
> However, It's only the c3 column which I can get.
> {code:java}
> +-+
> | _u1.c3 |
> +-+
> | c |
> | f |
> +-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993730#comment-16993730
]
Hive QA commented on HIVE-21164:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
19s{color} | {color:red} ql: The patch generated 45 new + 2685 unchanged - 24
fixed = 2730 total (was 2709) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} itests/hive-unit: The patch generated 22 new + 329
unchanged - 22 fixed = 351 total (was 351) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
48s{color} | {color:red} ql generated 1 new + 1531 unchanged - 1 fixed = 1532
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to filename in
org.apache.hadoop.hive.ql.io.AcidUtils.parseBucketId(Path) At
AcidUtils.java:org.apache.hadoop.hive.ql.io.AcidUtils.parseBucketId(Path) At
AcidUtils.java:[line 430] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19875/dev-support/hive-personality.sh
|
| git revision | master / eb72a0c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19875/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19875/yetus/diff-checkstyle-itests_hive-unit.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19875/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19875/yetus/patch-asflicense-problems.txt
|
| modules | C: ql hcatalog/streaming streaming itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19875/yetus.txt |
[jira] [Updated] (HIVE-22617) Re-Enable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[ https://issues.apache.org/jira/browse/HIVE-22617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22617: Status: Patch Available (was: Open) > Re-Enable PreCommit test > org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 > --- > > Key: HIVE-22617 > URL: https://issues.apache.org/jira/browse/HIVE-22617 > Project: Hive > Issue Type: Sub-task >Reporter: Oliver Draese >Assignee: László Bodor >Priority: Major > Attachments: HIVE-22617.01.patch, hive.log, initialize.log > > > The test was disabled via HIVE-22616 because it was flaky. If the test is > considered valid, it needs to be fixed and re-enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22601) Some columns will be lost when a UDTF has multiple aliases in some cases
[
https://issues.apache.org/jira/browse/HIVE-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993686#comment-16993686
]
Hive QA commented on HIVE-22601:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988500/HIVE-22601.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17755 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19874/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19874/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19874/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988500 - PreCommit-HIVE-Build
> Some columns will be lost when a UDTF has multiple aliases in some cases
>
>
> Key: HIVE-22601
> URL: https://issues.apache.org/jira/browse/HIVE-22601
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.1.2
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Attachments: HIVE-22601.1.patch, HIVE-22601.2.patch, HIVE-22601.patch
>
>
> Only one column will be retained when putting UDTFs with multiple aliases and
> a top-level UNION together.
> For example, the result of the following SQL should have three columns, c1,
> c2 and c3.
> {code:java}
> SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
> UNION ALL
> SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);
> {code}
> However, It's only the c3 column which I can get.
> {code:java}
> +-+
> | _u1.c3 |
> +-+
> | c |
> | f |
> +-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22594) Reorder metastore thrift structures
[ https://issues.apache.org/jira/browse/HIVE-22594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22594: Attachment: HIVE-22594.4.patch > Reorder metastore thrift structures > --- > > Key: HIVE-22594 > URL: https://issues.apache.org/jira/browse/HIVE-22594 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Thrift API >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22594.1.patch, HIVE-22594.2.patch, > HIVE-22594.3.patch, HIVE-22594.4.patch > > > Currently hive_metastore.thrift C++ output does not compile due to structures > like ScheduledQueryKey and ColumnStatistics being referenced before being > defined later in the file. We should reorder the structures in the file so > the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.7.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch, > HIVE-22572.6.patch, HIVE-22572.7.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22601) Some columns will be lost when a UDTF has multiple aliases in some cases
[
https://issues.apache.org/jira/browse/HIVE-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993635#comment-16993635
]
Hive QA commented on HIVE-22601:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
45s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 10 new + 278 unchanged - 2
fixed = 288 total (was 280) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19874/dev-support/hive-personality.sh
|
| git revision | master / eb72a0c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19874/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19874/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19874/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Some columns will be lost when a UDTF has multiple aliases in some cases
>
>
> Key: HIVE-22601
> URL: https://issues.apache.org/jira/browse/HIVE-22601
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.1.2
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Attachments: HIVE-22601.1.patch, HIVE-22601.2.patch, HIVE-22601.patch
>
>
> Only one column will be retained when putting UDTFs with multiple aliases and
> a top-level UNION together.
> For example, the result of the following SQL should have three columns, c1,
> c2 and c3.
> {code:java}
> SELECT stack(1, 'a', 'b', 'c') AS (c1, c2, c3)
> UNION ALL
> SELECT stack(1, 'd', 'e', 'f') AS (c1, c2, c3);
> {code}
> However, It's only the c3 column which I can get.
> {code:java}
> +-+
> | _u1.c3 |
> +-+
> | c |
> | f |
> +-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[
https://issues.apache.org/jira/browse/HIVE-22408?focusedWorklogId=357906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357906
]
ASF GitHub Bot logged work on HIVE-22408:
-
Author: ASF GitHub Bot
Created on: 11/Dec/19 15:31
Start Date: 11/Dec/19 15:31
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #860: HIVE-22408
The fix for CALCITE-2991 creates wrong results on edge case
URL: https://github.com/apache/hive/pull/860#discussion_r356660051
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitRemoveRule.java
##
@@ -41,8 +42,17 @@ public boolean matches(RelOptRuleCall call) {
final HiveSortLimit sortLimit = call.rel(0);
Double maxRowCount =
call.getMetadataQuery().getMaxRowCount(sortLimit.getInput());
-if (maxRowCount != null &&(maxRowCount <= 1)) {
- return true;
+if (maxRowCount != null) {
+ if (sortLimit.getFetchExpr() != null) {
+// we have LIMIT
+int limit = RexLiteral.intValue(sortLimit.getFetchExpr());
+if (maxRowCount <= limit) {
Review comment:
if I understand this correctly: this condition seems to be heavily building
on that `MaxRowCount` is an overestimation of the actual rowcount - at any time
that's not the case and it's an underestimation - we may end up removing the
limit.
Because of the above: I think it would make sense to add a toggle to
consider the removal in these cases or not...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 357906)
Time Spent: 0.5h (was: 20m)
> The fix for CALCITE-2991 creates wrong results on edge case
> ---
>
> Key: HIVE-22408
> URL: https://issues.apache.org/jira/browse/HIVE-22408
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch,
> HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This
> patch causes a failure in subquery_ALL.q
> In order to proceed with the upgrade, the patch was bypassed by creating the
> HiveRelMdMaxRowCount class which overrides the behavior in Calcite.
> In order to recreate the regression, remove this file and the references to
> the class in HiveDefaultRelMetadataProvider.
> This Jira is being created before the HIVE-22274 PR has been pushed, so
> please wait until that is pushed before working on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22408) The fix for CALCITE-2991 creates wrong results on edge case
[ https://issues.apache.org/jira/browse/HIVE-22408?focusedWorklogId=357905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357905 ] ASF GitHub Bot logged work on HIVE-22408: - Author: ASF GitHub Bot Created on: 11/Dec/19 15:31 Start Date: 11/Dec/19 15:31 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #860: HIVE-22408 The fix for CALCITE-2991 creates wrong results on edge case URL: https://github.com/apache/hive/pull/860#discussion_r356664585 ## File path: ql/src/test/results/clientpositive/llap/mapjoin_hint.q.out ## @@ -648,7 +651,7 @@ STAGE PLANS: outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 input vertices: 1 Reducer 3 -Statistics: Num rows: 1 Data size: 816 Basic stats: COMPLETE Column stats: NONE +Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Review comment: it's great that now we are able to deduce that there will be 0 rows... I'm wondering if there will be any sideeffects of this `Basic stats: NONE`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357905) Time Spent: 20m (was: 10m) > The fix for CALCITE-2991 creates wrong results on edge case > --- > > Key: HIVE-22408 > URL: https://issues.apache.org/jira/browse/HIVE-22408 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22408.1.patch, HIVE-22408.2.patch, > HIVE-22408.3.patch, HIVE-22408.4.patch, HIVE-22408.5.patch, HIVE-22408.6.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The Calcite upgrade from 1.19 to 1.21 includes a fix for CALCITE-2991. This > patch causes a failure in subquery_ALL.q > In order to proceed with the upgrade, the patch was bypassed by creating the > HiveRelMdMaxRowCount class which overrides the behavior in Calcite. > In order to recreate the regression, remove this file and the references to > the class in HiveDefaultRelMetadataProvider. > This Jira is being created before the HIVE-22274 PR has been pushed, so > please wait until that is pushed before working on this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993626#comment-16993626
]
Peter Vary commented on HIVE-22584:
---
The {{awaitTryScheduleIfInProgress}} waits for
{{TaskExecutorServiceForTest.tryScheduleUnderLock}} to finish. The waitQueue is
updated after the {{TaskExecutorService.tryScheduleUnderLock}} called in
{{WaitQueueWorker.run}}. Since the test checks the queue size, it is flaky
(~1/1000 just to make it easy :) )
The {{TaskExecutorService.tryScheduleUnderLock}} has the annotation of
{{VisibleForTesting}} and used only in a test, and in {{WaitQueueWorker.run}},
I think it is safe to assume that we can safely move the waitQueue changes
inside it. With this change I was able to run the tests ~4000 times without
failures.
[~kgyrtkirk] or [~szita], could you please review?
Thanks,
Peter
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.2.patch, HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-17680) TestNegativeCliDriver[merge_negative_5.q]
[
https://issues.apache.org/jira/browse/HIVE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-17680:
Status: Patch Available (was: Open)
> TestNegativeCliDriver[merge_negative_5.q]
> -
>
> Key: HIVE-17680
> URL: https://issues.apache.org/jira/browse/HIVE-17680
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-17680.01.branch-2.patch
>
>
> Seeing this error stack in the hive.log when I run it locally
> Test failed in
> https://builds.apache.org/job/PreCommit-HIVE-Build/7083/testReport
> {noformat}
> 2017-10-03T10:34:59,335 ERROR [d6419a0a-1fc9-4454-9575-f085c7f2cb55 main]
> metastore.RetryingHMSHandler: MetaException(message:The table must be
> bucketed and stored using an ACID compliant format (such as ORC))
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:199)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:68)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:53)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2290)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1413)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1511)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> com.sun.proxy.$Proxy34.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2407)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:760)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:748)
> at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:852)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:867)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4376)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1337)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1311)
> at
> org.apache.hadoop.hive.cli.control.CoreNegativeCliDriver.runTest(CoreNegativeCliDriver.java:121)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver(TestNegativeCliDriver.java:59)
> at
[jira] [Comment Edited] (HIVE-17680) TestNegativeCliDriver[merge_negative_5.q]
[
https://issues.apache.org/jira/browse/HIVE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993624#comment-16993624
]
László Bodor edited comment on HIVE-17680 at 12/11/19 3:20 PM:
---
I think this test is not 100% valid on branch-2 this way
the original exception was
{code}
The table must be bucketed and stored using an ACID compliant format (such as
ORC)
{code}
while creating this table
{code}
CREATE TABLE srcpart_acid (key STRING, value STRING) PARTITIONED BY (ds STRING,
hr STRING) stored as ORC TBLPROPERTIES ('transactional'='true',
'transactional_properties'='default');
{code}
seems like bucketing should be explicitly told while creating a transactional
table
after changing the CREATE TABLE ddl to use bucketing, the problem disappears,
but it also changes the negative output message from
{code}
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
Concatenate/Merge can not be performed on transactional tables
{code}
to
{code}
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
Concatenate/Merge can not be performed on bucketed tables
{code}
this q.out was only affected on branch-2 by HIVE-17403, so it should have been
failing since that
if we accept that expected negative output becomes "bucketed", this test could
remain valid, but a bit similar to merge_negative_4.q,merge_negative_5.q (which
was about bucketed table on master)
was (Author: abstractdog):
I thinks test is not 100% valid on branch-2
the original exception was
{code}
The table must be bucketed and stored using an ACID compliant format (such as
ORC)
{code}
while creating this table
{code}
CREATE TABLE srcpart_acid (key STRING, value STRING) PARTITIONED BY (ds STRING,
hr STRING) stored as ORC TBLPROPERTIES ('transactional'='true',
'transactional_properties'='default');
{code}
seems like bucketing should be explicitly told while creating a transactional
table
after changing the CREATE TABLE ddl to use bucketing, the probles disappears,
but it also changes the negative output message from
{code}
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
Concatenate/Merge can not be performed on transactional tables
{code}
to
{code}
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
Concatenate/Merge can not be performed on bucketed tables
{code}
this q.out was only affected on branch-2 by HIVE-17403, so it should have been
failing since that
if we accept that expected negative output becomes "bucketed", this test could
remain valid, but a bit similar or merge_negative_4.q,merge_negative_5.q (which
was about bucketed table on master)
> TestNegativeCliDriver[merge_negative_5.q]
> -
>
> Key: HIVE-17680
> URL: https://issues.apache.org/jira/browse/HIVE-17680
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-17680.01.branch-2.patch
>
>
> Seeing this error stack in the hive.log when I run it locally
> Test failed in
> https://builds.apache.org/job/PreCommit-HIVE-Build/7083/testReport
> {noformat}
> 2017-10-03T10:34:59,335 ERROR [d6419a0a-1fc9-4454-9575-f085c7f2cb55 main]
> metastore.RetryingHMSHandler: MetaException(message:The table must be
> bucketed and stored using an ACID compliant format (such as ORC))
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:199)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:68)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:53)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2290)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1413)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1511)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> com.sun.proxy.$Proxy34.create_table_with_environment_context(Unknown Source)
> at
>
[jira] [Commented] (HIVE-17680) TestNegativeCliDriver[merge_negative_5.q]
[
https://issues.apache.org/jira/browse/HIVE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993624#comment-16993624
]
László Bodor commented on HIVE-17680:
-
I thinks test is not 100% valid on branch-2
the original exception was
{code}
The table must be bucketed and stored using an ACID compliant format (such as
ORC)
{code}
while creating this table
{code}
CREATE TABLE srcpart_acid (key STRING, value STRING) PARTITIONED BY (ds STRING,
hr STRING) stored as ORC TBLPROPERTIES ('transactional'='true',
'transactional_properties'='default');
{code}
seems like bucketing should be explicitly told while creating a transactional
table
after changing the CREATE TABLE ddl to use bucketing, the probles disappears,
but it also changes the negative output message from
{code}
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
Concatenate/Merge can not be performed on transactional tables
{code}
to
{code}
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException:
Concatenate/Merge can not be performed on bucketed tables
{code}
this q.out was only affected on branch-2 by HIVE-17403, so it should have been
failing since that
if we accept that expected negative output becomes "bucketed", this test could
remain valid, but a bit similar or merge_negative_4.q,merge_negative_5.q (which
was about bucketed table on master)
> TestNegativeCliDriver[merge_negative_5.q]
> -
>
> Key: HIVE-17680
> URL: https://issues.apache.org/jira/browse/HIVE-17680
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-17680.01.branch-2.patch
>
>
> Seeing this error stack in the hive.log when I run it locally
> Test failed in
> https://builds.apache.org/job/PreCommit-HIVE-Build/7083/testReport
> {noformat}
> 2017-10-03T10:34:59,335 ERROR [d6419a0a-1fc9-4454-9575-f085c7f2cb55 main]
> metastore.RetryingHMSHandler: MetaException(message:The table must be
> bucketed and stored using an ACID compliant format (such as ORC))
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:199)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:68)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:53)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2290)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1413)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1511)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> com.sun.proxy.$Proxy34.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2407)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:760)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:748)
> at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:852)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:867)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4376)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
>
[jira] [Updated] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[
https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22584:
--
Attachment: HIVE-22584.2.patch
> Flakyness in TestTaskExecutorService.testSetCapacity
>
>
> Key: HIVE-22584
> URL: https://issues.apache.org/jira/browse/HIVE-22584
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22584.2.patch, HIVE-22584.patch
>
>
> Very rarely the test fails:
> {code}
> java.lang.AssertionError: expected:<0> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> See:
> https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-17680) TestNegativeCliDriver[merge_negative_5.q]
[
https://issues.apache.org/jira/browse/HIVE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-17680:
Attachment: HIVE-17680.01.branch-2.patch
> TestNegativeCliDriver[merge_negative_5.q]
> -
>
> Key: HIVE-17680
> URL: https://issues.apache.org/jira/browse/HIVE-17680
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-17680.01.branch-2.patch
>
>
> Seeing this error stack in the hive.log when I run it locally
> Test failed in
> https://builds.apache.org/job/PreCommit-HIVE-Build/7083/testReport
> {noformat}
> 2017-10-03T10:34:59,335 ERROR [d6419a0a-1fc9-4454-9575-f085c7f2cb55 main]
> metastore.RetryingHMSHandler: MetaException(message:The table must be
> bucketed and stored using an ACID compliant format (such as ORC))
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:199)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:68)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:53)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2290)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1413)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1511)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> com.sun.proxy.$Proxy34.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2407)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:760)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:748)
> at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:852)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:867)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4376)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1337)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1311)
> at
> org.apache.hadoop.hive.cli.control.CoreNegativeCliDriver.runTest(CoreNegativeCliDriver.java:121)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver(TestNegativeCliDriver.java:59)
>
[jira] [Assigned] (HIVE-17680) TestNegativeCliDriver[merge_negative_5.q]
[
https://issues.apache.org/jira/browse/HIVE-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reassigned HIVE-17680:
---
Assignee: László Bodor
> TestNegativeCliDriver[merge_negative_5.q]
> -
>
> Key: HIVE-17680
> URL: https://issues.apache.org/jira/browse/HIVE-17680
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: László Bodor
>Priority: Major
>
> Seeing this error stack in the hive.log when I run it locally
> Test failed in
> https://builds.apache.org/job/PreCommit-HIVE-Build/7083/testReport
> {noformat}
> 2017-10-03T10:34:59,335 ERROR [d6419a0a-1fc9-4454-9575-f085c7f2cb55 main]
> metastore.RetryingHMSHandler: MetaException(message:The table must be
> bucketed and stored using an ACID compliant format (such as ORC))
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleCreateTableTransactionalProp(TransactionalValidationListener.java:199)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.handle(TransactionalValidationListener.java:68)
> at
> org.apache.hadoop.hive.metastore.TransactionalValidationListener.onEvent(TransactionalValidationListener.java:53)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:2290)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1413)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1511)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> com.sun.proxy.$Proxy34.create_table_with_environment_context(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2407)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:760)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:748)
> at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:852)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:867)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4376)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1337)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1311)
> at
> org.apache.hadoop.hive.cli.control.CoreNegativeCliDriver.runTest(CoreNegativeCliDriver.java:121)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver(TestNegativeCliDriver.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
[jira] [Commented] (HIVE-22508) KeyWrapperComparator throws exception
[
https://issues.apache.org/jira/browse/HIVE-22508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993613#comment-16993613
]
Hive QA commented on HIVE-22508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12988495/HIVE-22508.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17770 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19873/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19873/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19873/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12988495 - PreCommit-HIVE-Build
> KeyWrapperComparator throws exception
> -
>
> Key: HIVE-22508
> URL: https://issues.apache.org/jira/browse/HIVE-22508
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22508.1.patch, HIVE-22508.1.patch,
> HIVE-22508.1.patch, HIVE-22508.2.patch, HIVE-22508.2.patch,
> HIVE-22508.2.patch, HIVE-22508.2.patch
>
>
> TopNKeyOperator.KeyWrapperComparator throws exception when a new key and a
> copied key should be compared.
> The current implementation uses the standard object inspectors for all
> KeyWrapper instances. However when comparing untouched KeyWrappers the key
> object inspector should be used which can be extracted form
> Operator.inputObjectInspectors during initialization of the key's
> ExprNodeEvaluator.
> This can cause a ClassCastException when the comparator is used collections
> like TreeSet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22604) TestMiniLlapCliDriver[table_nonprintable.q]
[ https://issues.apache.org/jira/browse/HIVE-22604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993607#comment-16993607 ] László Bodor commented on HIVE-22604: - pushed to branch-2, thanks for the review [~kgyrtkirk] > TestMiniLlapCliDriver[table_nonprintable.q] > --- > > Key: HIVE-22604 > URL: https://issues.apache.org/jira/browse/HIVE-22604 > Project: Hive > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-22604.01.branch-2.patch, > table_nonprintable.q.out.diff > > > q.out diff attached: [^table_nonprintable.q.out.diff] -- This message was sent by Atlassian Jira (v8.3.4#803005)
