[jira] [Commented] (HIVE-23082) PK/FK stat rescale doesn't work in some cases
[
https://issues.apache.org/jira/browse/HIVE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073385#comment-17073385
]
Hive QA commented on HIVE-23082:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21380/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21380/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PK/FK stat rescale doesn't work in some cases
> -
>
> Key: HIVE-23082
> URL: https://issues.apache.org/jira/browse/HIVE-23082
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23082.01.patch, HIVE-23082.02.patch,
> HIVE-23082.03.patch
>
>
> As a result in Joins may retain the original estimate; see MAPJOIN_33 in this
> plan ; which retained the estimate of SEL_32
> {code}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Map 1 <- Map 2 (BROADCAST_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:12 |
> | Stage-1|
> | Map 1 vectorized |
> | File Output Operator [FS_36]
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.17.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch, > HIVE-23004.14.patch, HIVE-23004.15.patch, HIVE-23004.16.patch, > HIVE-23004.17.patch, HIVE-23004.2.patch, HIVE-23004.4.patch, > HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch, > HIVE-23004.14.patch, HIVE-23004.15.patch, HIVE-23004.16.patch, > HIVE-23004.17.patch, HIVE-23004.2.patch, HIVE-23004.4.patch, > HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[
https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073377#comment-17073377
]
Hive QA commented on HIVE-23030:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998404/HIVE-23030.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18163 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21379/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21379/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21379/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998404 - PreCommit-HIVE-Build
> Enable sketch union-s to be rolled up
> -
>
> Key: HIVE-23030
> URL: https://issues.apache.org/jira/browse/HIVE-23030
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23030.01.patch, HIVE-23030.02.patch,
> HIVE-23030.03.patch, HIVE-23030.04.patch, HIVE-23030.04.patch,
> HIVE-23030.05.patch, HIVE-23030.06.patch, HIVE-23030.06.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Enabling rolling up sketch aggregates could enable the matching of
> materialized views created for higher dimensions to be applied for lower
> dimension cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22959) Extend storage-api to expose FilterContext
[
https://issues.apache.org/jira/browse/HIVE-22959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073359#comment-17073359
]
Jesus Camacho Rodriguez commented on HIVE-22959:
[~omalley], did you have the chance to look into this? {{storage-api}} release
is on hold until this is solved. Thanks
> Extend storage-api to expose FilterContext
> --
>
> Key: HIVE-22959
> URL: https://issues.apache.org/jira/browse/HIVE-22959
> Project: Hive
> Issue Type: Sub-task
> Components: storage-api
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, storage-2.7.2
>
> Attachments: HIVE-22959.1.patch, HIVE-22959.2.patch,
> HIVE-22959.3.patch, HIVE-22959.4.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> To enable row-level filtering at the ORC level ORC-577, or as an extension
> ProDecode MapJoin HIVE-22731 we need a common context class that will hold
> all the needed information for the filter.
> I propose this class to be part of the storage-api – similar to
> VectorizedRowBatch class and hold the information below:
> * A boolean variable showing if the filter is enabled
> * A int array storing the row Ids that are actually selected (passing the
> filter)
> * An int variable storing the the number or rows that passed the filter
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23128) SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
[
https://issues.apache.org/jira/browse/HIVE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23128:
--
Summary: SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
(was: SHOW CREATE TABLE Creates Invalid Syntax When Database Specified)
> SHOW CREATE TABLE Creates Incorrect Syntax When Database Specified
> --
>
> Key: HIVE-23128
> URL: https://issues.apache.org/jira/browse/HIVE-23128
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.4.0, 3.1.2
>Reporter: David Mollitor
>Priority: Major
>
> {code:sql}
> show create table `sample_07`;
> show create table `default`.`sample_07`;
> show create table `default.sample_07`;
> {code}
> {code:none|title=Results}
> CREATE TABLE `sample_07`(...)
> CREATE TABLE `default.sample_07`(...)
> CREATE TABLE `default.sample_07`(...);
> {code}
> All three queries complete in Hive 2.x (not sure about 3.x) but the results
> are consistent in both versions. The first result is correct because it does
> not include the database name, however, the subsequent two results are
> invalid: each field must be quoted individually.
> {quote}
> If any components of a multiple-part name require quoting, quote them
> individually rather than quoting the name as a whole. For example, write
> `my-table`.`my-column`, not `my-table.my-column`.
> * https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23030) Enable sketch union-s to be rolled up
[
https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073358#comment-17073358
]
Hive QA commented on HIVE-23030:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
36s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 174 unchanged - 0
fixed = 178 total (was 174) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21379/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21379/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21379/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21379/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enable sketch union-s to be rolled up
> -
>
> Key: HIVE-23030
> URL: https://issues.apache.org/jira/browse/HIVE-23030
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23030.01.patch, HIVE-23030.02.patch,
> HIVE-23030.03.patch, HIVE-23030.04.patch, HIVE-23030.04.patch,
> HIVE-23030.05.patch, HIVE-23030.06.patch, HIVE-23030.06.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Enabling rolling up sketch aggregates could enable the matching of
> materialized views created for higher dimensions to be applied for lower
> dimension cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-10775) Frequent calls to printStackTrace() obscuring legitimate problems
[ https://issues.apache.org/jira/browse/HIVE-10775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor resolved HIVE-10775. --- Resolution: Duplicate > Frequent calls to printStackTrace() obscuring legitimate problems > - > > Key: HIVE-10775 > URL: https://issues.apache.org/jira/browse/HIVE-10775 > Project: Hive > Issue Type: Improvement > Components: Metastore, Query Processor >Reporter: Andrew Cowie >Assignee: Andrew Cowie >Priority: Minor > Attachments: HIVE-10775.1.patch > > > When running test suites built on top of libraries that build on top of ... > that use Hive, the signal to noise ratio with exceptions flying past is > appalling. Most of this is down to calls to printStackTrace() embedded in > this library. HIVE-7697 showed someone cleaning that up and replacing with > logging the exception instead. That seems wise (logging can be redirected by > the calling test suite). > So, if you don't object, I'll hunt down the calls to printStackTrace() and > replace them with LOG.warn() instead. I'm about half way through the patch > now. > AfC -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073342#comment-17073342
]
Hive QA commented on HIVE-23004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998395/HIVE-23004.16.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18163 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17]
(batchId=158)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21378/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21378/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21378/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998395 - PreCommit-HIVE-Build
> Support Decimal64 operations across multiple vertices
> -
>
> Key: HIVE-23004
> URL: https://issues.apache.org/jira/browse/HIVE-23004
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch,
> HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.13.patch,
> HIVE-23004.14.patch, HIVE-23004.15.patch, HIVE-23004.16.patch,
> HIVE-23004.2.patch, HIVE-23004.4.patch, HIVE-23004.6.patch,
> HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch
>
>
> Support Decimal64 operations across multiple vertices
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: Patch Available (was: In Progress)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21492 started by Ganesha Shreedhara.
-
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: Open (was: Patch Available)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073324#comment-17073324
]
Ganesha Shreedhara commented on HIVE-21492:
---
Test failures are unrelated and mostly because the metstore server was down.
{code:java}
Could not connect to meta store using any of the URIs provided. Most recent
failure: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused at
org.apache.thrift.transport.TSocket.open(TSocket.java:226) at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:686){code}
Can we rerun the tests?
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073320#comment-17073320
]
Hive QA commented on HIVE-23004:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} serde: The patch generated 7 new + 713 unchanged - 1
fixed = 720 total (was 714) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 3 new + 410 unchanged - 1
fixed = 413 total (was 411) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red} serde generated 4 new + 194 unchanged - 3 fixed = 198
total (was 197) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} ql generated 0 new + 1527 unchanged - 1 fixed = 1527
total (was 1528) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNotNullMarker At
BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal
representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNotNullMarker At
BinarySortableDeserializeRead.java:[line 151] |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNullMarker At
BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal
representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNullMarker At
BinarySortableDeserializeRead.java:[line 150] |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnSortOrderIsDesc At
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073305#comment-17073305
]
Hive QA commented on HIVE-21492:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998415/HIVE-21492.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 18162 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=252)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.alterTableBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.createTableInBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.dropTableBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.getAllTablesInBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.getMaterializedViewsInBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.getTableInBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.getTableObjectsByNameBogusCatalog[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.moveTablesBetweenCatalogsOnAlter[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.tablesInOtherCatalogs[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableAlreadyExists[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableCascade[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableChangeCols[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableChangingDatabase[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableEmptyTableNameInNew[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableExternalTableChangeLocation[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableExternalTable[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorAddPartitionColumns[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorAlterPartitionColumnName[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorInvalidColumnType[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorNullCols[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorNullColumnType[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorNullLocation[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorNullSerdeInfo[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidStorageDescriptorRemovePartitionColumn[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableInvalidTableNameInNew[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNoSuchDatabase[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNoSuchTableInThisDatabase[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNoSuchTable[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullDatabaseInNew[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullDatabase[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullNewTable[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Remote]
(batchId=230)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073278#comment-17073278
]
Hive QA commented on HIVE-21492:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21377/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21377/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
>
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073277#comment-17073277
]
Ferdinand Xu commented on HIVE-21492:
-
LGTM +1 pending on the test
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073270#comment-17073270
]
Hive QA commented on HIVE-23118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998390/HIVE-23118.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18163 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=15)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_compile_counters]
(batchId=186)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21376/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21376/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21376/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998390 - PreCommit-HIVE-Build
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23126: Attachment: HIVE-23126.2.patch Status: Patch Available (was: Open) > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23126.1.patch, HIVE-23126.2.patch > > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23126: Status: Open (was: Patch Available) > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23126.1.patch > > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23127) Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner
[ https://issues.apache.org/jira/browse/HIVE-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-23127: --- Status: Patch Available (was: Open) > Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner > > > Key: HIVE-23127 > URL: https://issues.apache.org/jira/browse/HIVE-23127 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-23127.1.patch > > > GetPartitionsWithSpecs reduces data transfer by deduplicating storage > descriptor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23127) Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner
[ https://issues.apache.org/jira/browse/HIVE-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-23127: --- Attachment: HIVE-23127.1.patch > Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner > > > Key: HIVE-23127 > URL: https://issues.apache.org/jira/browse/HIVE-23127 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-23127.1.patch > > > GetPartitionsWithSpecs reduces data transfer by deduplicating storage > descriptor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23127) Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner
[ https://issues.apache.org/jira/browse/HIVE-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-23127: -- > Replace listPartitionsByExpr with GetPartitionsWithSpecs in Partition pruner > > > Key: HIVE-23127 > URL: https://issues.apache.org/jira/browse/HIVE-23127 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > GetPartitionsWithSpecs reduces data transfer by deduplicating storage > descriptor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[
https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073248#comment-17073248
]
Hive QA commented on HIVE-23118:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 4 new + 88 unchanged - 0 fixed
= 92 total (was 88) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21376/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21376/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21376/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Option for exposing compile time counters as tez counters
> -
>
> Key: HIVE-23118
> URL: https://issues.apache.org/jira/browse/HIVE-23118
> Project: Hive
> Issue Type: Improvement
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23118.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TezCounters currently are runtime only. Some compile time information from
> optimizer can be exposed as counters which can then be used by workload
> management to make runtime decisions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21328) Call To Hadoop Text getBytes() Without Call to getLength()
[
https://issues.apache.org/jira/browse/HIVE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21328:
--
Status: Patch Available (was: Open)
> Call To Hadoop Text getBytes() Without Call to getLength()
> --
>
> Key: HIVE-21328
> URL: https://issues.apache.org/jira/browse/HIVE-21328
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21328.1.patch
>
>
> I'm not sure if there is actually a bug, but this looks highly suspect:
> {code:java}
> public Object set(final Object o, final Text text) {
> return new BytesWritable(text == null ? null : text.getBytes());
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/primitive/ParquetStringInspector.java#L104-L106
> There are two components to a Text object. There are the internal bytes and
> the length of the bytes. The two are independent. I.e., a quick "reset" on
> the Text object simply sets the internal length counter to zero. This code
> is potentially looking at obsolete data that it shouldn't be seeing because
> it is not considering the length of the Text.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23117) Review of HiveStatement Class
[
https://issues.apache.org/jira/browse/HIVE-23117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073230#comment-17073230
]
Hive QA commented on HIVE-23117:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998383/HIVE-23117.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21375/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21375/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21375/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998383 - PreCommit-HIVE-Build
> Review of HiveStatement Class
> -
>
> Key: HIVE-23117
> URL: https://issues.apache.org/jira/browse/HIVE-23117
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23117.1.patch
>
>
> * Remove unused instance variable(s)
> * Remove non-JavaDoc comments
> * Make inPlaceUpdateStream Optional (and remove NO-OP class) (inconsistent
> behavior with 'null' values)
> * {{getQueryTimeout()}} returns incorrect value
> * Unify and improve Exception messages
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23096) Review Code Path for getResults
[
https://issues.apache.org/jira/browse/HIVE-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23096:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~ngangam] for the review!
> Review Code Path for getResults
> ---
>
> Key: HIVE-23096
> URL: https://issues.apache.org/jira/browse/HIVE-23096
> Project: Hive
> Issue Type: Sub-task
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23096.1.patch
>
>
> * Log when a buffer returned to a client is completely full. This can be
> helpful for identifying misconfigured clients that have a fetchSize that is
> too small
> * See below code snippet
> * Rows are loaded into an ArrayList called "convey." This ArrayList is
> instantiated with the default size (10) and is expanded on-demand as it is
> filled. I changed it so that this ArrayList's size is set once, on the first
> call to fetch, so that it doesn't have to "grow" and waste cycles and memory
> with GC thrashing for that.
> * Some minor cleanup
> {code:java|Driver.java}
> try {
> ss = Utilities.readColumn(driverContext.getResStream(), bos);
> if (bos.getLength() > 0) {
> // use JDK StandardCharsets
> row = new String(bos.getData(), 0, bos.getLength(), "UTF-8");
> } else if (ss == Utilities.StreamStatus.TERMINATED) {
> // Do not create a new string for this. Just use a hard-coded
> empty string. JDK: "Note that use of this constructor is unnecessary since
> Strings are immutable."
> row = new String();
> }
> ...
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23117) Review of HiveStatement Class
[
https://issues.apache.org/jira/browse/HIVE-23117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073198#comment-17073198
]
Hive QA commented on HIVE-23117:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} jdbc: The patch generated 0 new + 6 unchanged - 1
fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21375/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21375/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of HiveStatement Class
> -
>
> Key: HIVE-23117
> URL: https://issues.apache.org/jira/browse/HIVE-23117
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23117.1.patch
>
>
> * Remove unused instance variable(s)
> * Remove non-JavaDoc comments
> * Make inPlaceUpdateStream Optional (and remove NO-OP class) (inconsistent
> behavior with 'null' values)
> * {{getQueryTimeout()}} returns incorrect value
> * Unify and improve Exception messages
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
[ https://issues.apache.org/jira/browse/HIVE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073191#comment-17073191 ] Jesus Camacho Rodriguez commented on HIVE-23106: +1 > Cleanup CalcitePlanner genOPTree exception handling > --- > > Key: HIVE-23106 > URL: https://issues.apache.org/jira/browse/HIVE-23106 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Minor > Attachments: HIVE-23106.1.patch, HIVE-23106.2.patch > > > The logic where genOPTree handles exceptions during CBO planning is a bit > twisty and could use some cleanup and comments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
[
https://issues.apache.org/jira/browse/HIVE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073189#comment-17073189
]
Hive QA commented on HIVE-23106:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998377/HIVE-23106.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21374/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21374/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21374/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998377 - PreCommit-HIVE-Build
> Cleanup CalcitePlanner genOPTree exception handling
> ---
>
> Key: HIVE-23106
> URL: https://issues.apache.org/jira/browse/HIVE-23106
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Attachments: HIVE-23106.1.patch, HIVE-23106.2.patch
>
>
> The logic where genOPTree handles exceptions during CBO planning is a bit
> twisty and could use some cleanup and comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
[
https://issues.apache.org/jira/browse/HIVE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073159#comment-17073159
]
Hive QA commented on HIVE-23106:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 115 unchanged - 7
fixed = 115 total (was 122) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21374/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21374/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cleanup CalcitePlanner genOPTree exception handling
> ---
>
> Key: HIVE-23106
> URL: https://issues.apache.org/jira/browse/HIVE-23106
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Attachments: HIVE-23106.1.patch, HIVE-23106.2.patch
>
>
> The logic where genOPTree handles exceptions during CBO planning is a bit
> twisty and could use some cleanup and comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23126: Attachment: HIVE-23126.1.patch Status: Patch Available (was: Open) > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23126.1.patch > > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-23126: --- > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23094) Implement Explain CBO of Update and Delete statements
[
https://issues.apache.org/jira/browse/HIVE-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073141#comment-17073141
]
Hive QA commented on HIVE-23094:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998369/HIVE-23094.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21373/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21373/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21373/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998369 - PreCommit-HIVE-Build
> Implement Explain CBO of Update and Delete statements
> -
>
> Key: HIVE-23094
> URL: https://issues.apache.org/jira/browse/HIVE-23094
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23094.1.patch
>
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as
> orc TBLPROPERTIES ('transactional'='true');
> explain cbo
> update acidtlb set b=777;
> {code}
> doesn't print CBO plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21328) Call To Hadoop Text getBytes() Without Call to getLength()
[
https://issues.apache.org/jira/browse/HIVE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21328:
--
Attachment: HIVE-21328.1.patch
> Call To Hadoop Text getBytes() Without Call to getLength()
> --
>
> Key: HIVE-21328
> URL: https://issues.apache.org/jira/browse/HIVE-21328
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21328.1.patch
>
>
> I'm not sure if there is actually a bug, but this looks highly suspect:
> {code:java}
> public Object set(final Object o, final Text text) {
> return new BytesWritable(text == null ? null : text.getBytes());
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/primitive/ParquetStringInspector.java#L104-L106
> There are two components to a Text object. There are the internal bytes and
> the length of the bytes. The two are independent. I.e., a quick "reset" on
> the Text object simply sets the internal length counter to zero. This code
> is potentially looking at obsolete data that it shouldn't be seeing because
> it is not considering the length of the Text.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-21328) Call To Hadoop Text getBytes() Without Call to getLength()
[
https://issues.apache.org/jira/browse/HIVE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-21328:
-
Assignee: David Mollitor
> Call To Hadoop Text getBytes() Without Call to getLength()
> --
>
> Key: HIVE-21328
> URL: https://issues.apache.org/jira/browse/HIVE-21328
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21328.1.patch
>
>
> I'm not sure if there is actually a bug, but this looks highly suspect:
> {code:java}
> public Object set(final Object o, final Text text) {
> return new BytesWritable(text == null ? null : text.getBytes());
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/primitive/ParquetStringInspector.java#L104-L106
> There are two components to a Text object. There are the internal bytes and
> the length of the bytes. The two are independent. I.e., a quick "reset" on
> the Text object simply sets the internal length counter to zero. This code
> is potentially looking at obsolete data that it shouldn't be seeing because
> it is not considering the length of the Text.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: (was: HIVE-23067.3.patch) > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.4.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.3.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.3.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23094) Implement Explain CBO of Update and Delete statements
[
https://issues.apache.org/jira/browse/HIVE-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073085#comment-17073085
]
Hive QA commented on HIVE-23094:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21373/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21373/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Explain CBO of Update and Delete statements
> -
>
> Key: HIVE-23094
> URL: https://issues.apache.org/jira/browse/HIVE-23094
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23094.1.patch
>
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as
> orc TBLPROPERTIES ('transactional'='true');
> explain cbo
> update acidtlb set b=777;
> {code}
> doesn't print CBO plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073058#comment-17073058
]
Hive QA commented on HIVE-23039:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998454/HIVE-23039.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 18168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointing
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointingInDumpFailure
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=283)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointing
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointingInDumpFailure
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableIncrementalReplication
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableReplicationWithCustomPaths
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableWithPartitions
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21372/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21372/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21372/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998454 - PreCommit-HIVE-Build
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch,
> HIVE-23039.03.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.03.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-18897) Hive is Double-Logging Invalid UDF Error
[
https://issues.apache.org/jira/browse/HIVE-18897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-18897.
---
Resolution: Not A Problem
Based on the current master branch, it looks like the logging was fixed to
remove the ERROR log in the {{Driver}} class.
> Hive is Double-Logging Invalid UDF Error
>
>
> Key: HIVE-18897
> URL: https://issues.apache.org/jira/browse/HIVE-18897
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0
>Reporter: David Mollitor
>Assignee: Ramesh Kumar Thangarajan
>Priority: Minor
> Labels: newbie, noob
>
> It logs the "invalid" function twice. Once at ERROR level and once at WARN
> level. Please change so that the error is logged once, at the WARN level.
> The stack trace seems overkill here as well for such a trivial error...
> usually a user typo or the function needs to be registered.
> {code:java}
> 2018-03-05 07:50:44,473 ERROR org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-43]: FAILED: SemanticException [Error
> 10011]: Line 1:7 Invalid function 'aes_encrypt'
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:7 Invalid function
> 'aes_encrypt'
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:836)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10422)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10378)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3771)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3550)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8830)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8785)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9652)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9545)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10018)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10029)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9909)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1261)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:143)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:215)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:402)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
[jira] [Resolved] (HIVE-18956) AvroSerDe Race Condition
[
https://issues.apache.org/jira/browse/HIVE-18956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-18956.
---
Resolution: Won't Fix
> AvroSerDe Race Condition
>
>
> Key: HIVE-18956
> URL: https://issues.apache.org/jira/browse/HIVE-18956
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0, 2.3.2
>Reporter: David Mollitor
>Priority: Trivial
>
> {code}
> @Override
> public Writable serialize(Object o, ObjectInspector objectInspector) throws
> SerDeException {
> if(badSchema) {
> throw new BadSchemaException();
> }
> return getSerializer().serialize(o, objectInspector, columnNames,
> columnTypes, schema);
> }
> @Override
> public Object deserialize(Writable writable) throws SerDeException {
> if(badSchema) {
> throw new BadSchemaException();
> }
> return getDeserializer().deserialize(columnNames, columnTypes, writable,
> schema);
> }
> ...
> private AvroDeserializer getDeserializer() {
> if(avroDeserializer == null) {
> avroDeserializer = new AvroDeserializer();
> }
> return avroDeserializer;
> }
> private AvroSerializer getSerializer() {
> if(avroSerializer == null) {
> avroSerializer = new AvroSerializer();
> }
> return avroSerializer;
> }
> {code}
> {{getDeserializer}} and {{getSerializer}} methods are not thread safe, so
> neither are {{deserialize}} and {{serialize}} methods. It probably didn't
> matter with MapReduce, but now that we have Spark/Tez, it may be an issue.
> You could visualize a scenario where three threads all enter
> {{getSerializer}} and all see that {{avroSerializer}} is _null_ and create
> three instances, then they would fight to assign the new object to the
> {{avroSerializer}} variable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073001#comment-17073001
]
Hive QA commented on HIVE-23039:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 9 new + 59 unchanged - 0 fixed
= 68 total (was 59) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 688
unchanged - 0 fixed = 690 total (was 688) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21372/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21372/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21372/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21372/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23114) Insert overwrite with dynamic partitioning is not working correctly with direct insert
[
https://issues.apache.org/jira/browse/HIVE-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072955#comment-17072955
]
Hive QA commented on HIVE-23114:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998356/HIVE-23114.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18166 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21371/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21371/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21371/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998356 - PreCommit-HIVE-Build
> Insert overwrite with dynamic partitioning is not working correctly with
> direct insert
> --
>
> Key: HIVE-23114
> URL: https://issues.apache.org/jira/browse/HIVE-23114
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-23114.1.patch
>
>
> This is a follow-up Jira for the
> [conversation|https://issues.apache.org/jira/browse/HIVE-21164?focusedCommentId=17059280=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17059280]
> in HIVE-21164
> Doing an insert overwrite from a multi-insert statement with dynamic
> partitioning will give wrong results for ACID tables when
> 'hive.acid.direct.insert.enabled' is true or for insert-only tables.
> Reproduction:
> {noformat}
> set hive.acid.direct.insert.enabled=true;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.vectorized.execution.enabled=false;
> set hive.stats.autogather=false;
> create external table multiinsert_test_text (a int, b int, c int) stored as
> textfile;
> insert into multiinsert_test_text values (, 11, ), (, 22, ),
> (, 33, ), (, 44, NULL), (, 55, NULL);
> create table multiinsert_test_acid (a int, b int) partitioned by (c int)
> stored as orc tblproperties('transactional'='true');
> create table multiinsert_test_mm (a int, b int) partitioned by (c int) stored
> as orc tblproperties('transactional'='true',
> 'transactional_properties'='insert_only');
> from multiinsert_test_text a
> insert overwrite table multiinsert_test_acid partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is not null
> insert overwrite table multiinsert_test_acid partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is null;
> select * from multiinsert_test_acid;
> from multiinsert_test_text a
> insert overwrite table multiinsert_test_mm partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is not null
> insert overwrite table multiinsert_test_mm partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is null;
> select * from multiinsert_test_mm;
> {noformat}
> The result of these steps can be different, it depends on the execution order
> of the FileSinkOperators of the insert overwrite statements. It can happen
> that an error occurs due to manifest file collision, it can happen that no
> error occurs but the result will be incorrect.
> Running the same insert query with an external table of with and ACID table
> with 'hive.acid.direct.insert.enabled=false' will give the follwing result:
> {noformat}
> 11
> 22
> 33
> 44 NULL
> 55 NULL
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23114) Insert overwrite with dynamic partitioning is not working correctly with direct insert
[
https://issues.apache.org/jira/browse/HIVE-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072883#comment-17072883
]
Hive QA commented on HIVE-23114:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 1 new + 314 unchanged - 0
fixed = 315 total (was 314) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
54s{color} | {color:red} ql generated 1 new + 1528 unchanged - 0 fixed = 1529
total (was 1528) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field
org.apache.hadoop.hive.ql.exec.FileSinkOperator.dynamicPartitionSpecs is
transient but isn't set by deserialization In FileSinkOperator.java:but isn't
set by deserialization In FileSinkOperator.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21371/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21371/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21371/yetus/new-findbugs-ql.html
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21371/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Insert overwrite with dynamic partitioning is not working correctly with
> direct insert
> --
>
> Key: HIVE-23114
> URL: https://issues.apache.org/jira/browse/HIVE-23114
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-23114.1.patch
>
>
> This is a follow-up Jira for the
> [conversation|https://issues.apache.org/jira/browse/HIVE-21164?focusedCommentId=17059280=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17059280]
> in HIVE-21164
> Doing an insert overwrite from a multi-insert statement with dynamic
> partitioning will give wrong results for ACID tables
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072869#comment-17072869
]
László Pintér commented on HIVE-23109:
--
+1
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072850#comment-17072850
]
Hive QA commented on HIVE-23109:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998353/HIVE-23109.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18166 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21370/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21370/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21370/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998353 - PreCommit-HIVE-Build
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072821#comment-17072821
]
László Bodor commented on HIVE-23122:
-
cc: [~rajesh.balamohan]
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23098) Allow Operation assertState to Accept a Collection
[
https://issues.apache.org/jira/browse/HIVE-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072814#comment-17072814
]
David Mollitor commented on HIVE-23098:
---
[~ngangam] Here is another trivial review if you have any spare cycles (lol).
> Allow Operation assertState to Accept a Collection
> --
>
> Key: HIVE-23098
> URL: https://issues.apache.org/jira/browse/HIVE-23098
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23098.1.patch, HIVE-23098.2.patch,
> HIVE-23098.2.patch, HIVE-23098.2.patch, HIVE-23098.3.patch
>
>
> {code:java|title=Operation.java}
> protected final void assertState(List states) throws
> HiveSQLException {
> if (!states.contains(state)) {
> throw new HiveSQLException("Expected states: " + states.toString() + ",
> but found "
> + this.state);
> }
> this.lastAccessTime = System.currentTimeMillis();
> }
> /*/
> public void someMethod() {
> assertState(new
> ArrayList(Arrays.asList(OperationState.FINISHED)));
> }
> {code}
> By allowing {{assertState}} to accept a {{Collection}}, one can save an
> allocation and simplify the code:
> {code:java}
> assertState(Collections.singleton(OperationState.FINISHED));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-22484.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23124:
--
Description:
* Use ConcurrentHashMap instead of synchronized methods to improve
multi-threaded access
* Use JDK 8 facilities where applicable
* General cleanup
* Better log messages and Exception messages
* Use {{switch}} statement instead of if/else blocks
* Checkstyle fixes
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23124: -- Attachment: HIVE-23124.1.patch > Review of SQLOperation Class > > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Attachment: HIVE-23089.1.patch
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Status: Patch Available (was: Open)
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23124) Review of SQLOperation
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23124: - Assignee: David Mollitor > Review of SQLOperation > --- > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23124: -- Summary: Review of SQLOperation Class (was: Review of SQLOperation ) > Review of SQLOperation Class > > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23124: -- Status: Patch Available (was: Open) > Review of SQLOperation Class > > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072809#comment-17072809
]
Hive QA commented on HIVE-23109:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21370/dev-support/hive-personality.sh
|
| git revision | master / 484d182 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21370/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072785#comment-17072785
]
Panagiotis Garefalakis commented on HIVE-23122:
---
Thanks [~abstractdog]
+1 tests pending
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Summary: LLAP: TaskExecutorService should log details about task eviction
decision details (was: LLAP TaskExecutorService should log details about task
eviction decision details)
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072783#comment-17072783
]
László Bodor commented on HIVE-23122:
-
thanks for the comments [~pgaref], included canFinishForPriority
removing 01.patch for saving ptest time
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Attachment: (was: HIVE-23122.01.patch)
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Attachment: HIVE-23122.02.patch
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072781#comment-17072781
]
Hive QA commented on HIVE-23107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998338/HIVE-23107.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 160 failed/errored test(s), 18162 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindNextToClean
(batchId=318)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned
(batchId=318)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents
(batchId=318)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCompacted
(batchId=318)
org.apache.hadoop.hive.ql.TestAcidOnTez.testInsertWithRemoveUnion (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnMR (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnMR (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnTez (batchId=253)
org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01
(batchId=363)
org.apache.hadoop.hive.ql.TestTxnCommands.testVersioning (batchId=363)
org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolutionAndCompaction
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testCleanerForTxnToWriteId
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testCompactWithDelete (batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testEmptyInTblproperties
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite1 (batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite2 (batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testSchemaEvolutionCompaction
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.writeBetweenWorkerAndCleaner
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testACIDwithSchemaEvolutionAndCompaction
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCleanerForTxnToWriteId
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCompactWithDelete
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testEmptyInTblproperties
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInitiatorWithMultipleFailedCompactions
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite1
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite2
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testSchemaEvolutionCompaction
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.writeBetweenWorkerAndCleaner
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands3.testAcidMetaColumsDecode
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testCleaner2 (batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testCompactionAbort (batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOff
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOn
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testNotEnoughToCompactNeedsCleaning
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=318)
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of
[jira] [Commented] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072779#comment-17072779
]
Miklos Gergely commented on HIVE-23123:
---
[~jcamachorodriguez] please let me know what you think.
> Export/Import of views and materialized views
> -
>
> Key: HIVE-23123
> URL: https://issues.apache.org/jira/browse/HIVE-23123
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
> import and export can be done by using the
>
> {code:java}
> export table ...
> import table ...
> {code}
> commands. The document doesn't mention views or materialized views at all,
> and in fact we don't support commands like
>
> {code:java}
> export view ...
> import view ...
> export materialized view ...
> import materialized view ...
> {code}
> they can not be parsed at all. The word table is often used though in a
> broader sense, when it means all table like entities, including views and
> materialized views. For example the various Table classes may represent any
> of these as well.
> If I try to export a view with the export table ... command, it goes fine. A
> _metadata file will be created, but no data directory, which is what we'd
> expect. If I try to import it back, an exception is thrown due to the lack of
> the data dir:
>
> {code:java}
> java.lang.AssertionError: null==getPath() for exim_view
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
> {code}
> Still the view gets imported successfully, as data movement wasn't even
> necessary.
> If we try to export a materialized view which is transactional, then this
> exception occurs:
>
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
> at
> org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
> {code}
> So the export process can not handle it, as the temporary table is not
> getting created.
>
> The import command handling have a lot of codes dedicated to importing views
> and materialized views, which suggests that we support the importing (and
> thus also suggests implicitly that we support the exporting) of views and
> materialiezed views.
>
> So the conclusion is that we have to decide if we support exporting/importing
> of views and materialized views.
> If we decide not to support them then:
> - export process should throw an exception if a view or materialized view is
> the subject
> - the codes specific to view imports should be removed
> If we decide to support them, then:
> - the commands mentioned above above should be introduced
> - exception should be thrown if not the proper command used (e.g. export
> view on a table)
> - the exceptions mentioned above should be fixed
> I prefer not to support them, I don't think we should support the exporting /
> importing of views. The point of exporting / importing is the transfer of
> data, not DDL, it causes more issues than it solves. Our current
> documentation also suggests that it is only supported for tables.
>
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
# If we decide not to support them then:
#
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
# If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer #1, I don't think we should support the exporting / importing of
views. The point of exporting / importing is the transfer of data, not DDL, it
causes more issues than it solves. Our current documentation also suggests that
it is only supported for tables.
was:
According to
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer #1, I don't think we should support the exporting / importing of
views. The point of exporting / importing is the transfer of data, not DDL, it
causes more issues than it solves. Our current documentation also suggests that
it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any
[jira] [Assigned] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely reassigned HIVE-23123:
-
> Export/Import of views and materialized views
> -
>
> Key: HIVE-23123
> URL: https://issues.apache.org/jira/browse/HIVE-23123
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> According to
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
> import and export can be done by using the
>
> {code:java}
> export table ...
> import table ...
> {code}
> commands. The document doesn't mention views or materialized views at all,
> and in fact we don't support commands like
>
> {code:java}
> export view ...
> import view ...
> export materialized view ...
> import materialized view ...
> {code}
> they can not be parsed at all. The word table is often used though in a
> broader sense, when it means all table like entities, including views and
> materialized views. For example the various Table classes may represent any
> of these as well.
> If I try to export a view with the export table ... command, it goes fine. A
> _metadata file will be created, but no data directory, which is what we'd
> expect. If I try to import it back, an exception is thrown due to the lack of
> the data dir:
>
> {code:java}
> java.lang.AssertionError: null==getPath() for exim_view
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
> {code}
> Still the view gets imported successfully, as data movement wasn't even
> necessary.
> If we try to export a materialized view which is transactional, then this
> exception occurs:
>
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
> at
> org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
> {code}
> So the export process can not handle it, as the temporary table is not
> getting created.
>
> The import command handling have a lot of codes dedicated to importing views
> and materialized views, which suggests that we support the importing (and
> thus also suggests implicitly that we support the exporting) of views and
> materialiezed views.
>
> So the conclusion is that we have to decide if we support exporting/importing
> of views and materialized views.
> # If we decide not to support them then:
> - export process should throw an exception if a view or materialized view is
> the subject
> - the codes specific to view imports should be removed
> # If we decide to support them, then:
> - the commands mentioned above above should be introduced
> - exception should be thrown if not the proper command used (e.g. export
> view on a table)
> - the exceptions mentioned above should be fixed
> I prefer #1, I don't think we should support the exporting / importing of
> views. The point of exporting / importing is the transfer of data, not DDL,
> it causes more issues than it solves. Our current documentation also suggests
> that it is only supported for tables.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23079) Remove Calls to printStackTrace in Module hive-serde
[ https://issues.apache.org/jira/browse/HIVE-23079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23079: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~pvary] for the review! > Remove Calls to printStackTrace in Module hive-serde > > > Key: HIVE-23079 > URL: https://issues.apache.org/jira/browse/HIVE-23079 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23079.1.patch, HIVE-23079.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23113:
--
Attachment: HIVE-23113.1.patch
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch, HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072736#comment-17072736
]
Hive QA commented on HIVE-23107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 4 new + 669 unchanged - 2 fixed = 673 total (was 671) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 187 unchanged - 3 fixed = 187 total (was 190) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} ql in the patch passed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore_metastore-server generated 1 new
+ 24 unchanged - 0 fixed = 25 total (was 24) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21369/dev-support/hive-personality.sh
|
| git revision | master / 94f90bc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21369/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21369/yetus/diff-javadoc-javadoc-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21369/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove MIN_HISTORY_LEVEL table
> --
>
> Key: HIVE-23107
> URL: https://issues.apache.org/jira/browse/HIVE-23107
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-23107.01.patch, HIVE-23107.02.patch
>
>
> MIN_HISTORY_LEVEL table is used in two places:
> * Cleaner uses it to decide if the files can
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.02.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072716#comment-17072716
]
Panagiotis Garefalakis commented on HIVE-23122:
---
Thanks for the patch [~abstractdog] – pretty useful – would it make sense to
add the canFinishForPriority property used by the BaseComparator as well?
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.01.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072699#comment-17072699
]
Hive QA commented on HIVE-23113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998347/HIVE-23113.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18162 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testRowsAffected (batchId=286)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998347 - PreCommit-HIVE-Build
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Description:
TaskExecutorService maintains a waitQueue, and can evict a task in favor of
another. Under the hood, the queue uses a configurable
[comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
The currently available comparators typically use the following properties of
a task(wrapper):
getWithinDagPriority: related to vertex
currentAttemptStartTime
firstAttemptStartTime
knownPending: remaining upstream tasks
The problem is, when an eviction happens, the INFO level message doesn't
provide any insight about the decision, only attempts ids like below:
{code}
attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
attempt_1585248378306_0003_175_02_79_175 because of lower priority
{code}
was:
TaskExecutorService maintains a waitQueue, and can evict a task in favor of
another. Under the hood, the queue uses a configurable
[comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
The currently available comparators typically use the following properties of
a task:
getWithinDagPriority: related to vertex
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.01.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Description: TaskExecutorService maintains a waitQueue, and can evict a task in favor of another. Under the hood, the queue uses a configurable [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. The currently available comparators typically use the following properties of a task: getWithinDagPriority: related to vertex > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task: > getWithinDagPriority: related to vertex -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Status: Patch Available (was: Open) > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Attachment: HIVE-23122.01.patch > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Summary: LLAP TaskExecutorService should log details about task eviction decision details (was: LLAP TaskSchedulerService should log details about task eviction decision details) > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23122) LLAP TaskSchedulerService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-23122: --- Assignee: László Bodor > LLAP TaskSchedulerService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.20.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23121) Re-examine TestWarehouseExternalDir to see if it uses HMS translation.
[ https://issues.apache.org/jira/browse/HIVE-23121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-23121: > Re-examine TestWarehouseExternalDir to see if it uses HMS translation. > -- > > Key: HIVE-23121 > URL: https://issues.apache.org/jira/browse/HIVE-23121 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > TestWarehouseExternalDir currently passes with just one change related to > HIVE-22995. But that change was assuming it was using HMS Translation to > convert non-acid managed table to external. > Ensure that it still does. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072649#comment-17072649
]
Hive QA commented on HIVE-23113:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} jdbc: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21368/dev-support/hive-personality.sh
|
| git revision | master / 94f90bc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21368/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23020) Avoid using _files for replication data copy during incremental run
[
https://issues.apache.org/jira/browse/HIVE-23020?focusedWorklogId=413976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413976
]
ASF GitHub Bot logged work on HIVE-23020:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:51
Start Date: 01/Apr/20 10:51
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #963: HIVE-23020 :
Avoid using _files for replication data copy during incr…
URL: https://github.com/apache/hive/pull/963#discussion_r401526351
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AddPartitionHandler.java
##
@@ -108,10 +108,8 @@ public void handle(Context withinContext) throws
Exception {
Iterable files = partitionFilesIter.next().getFiles();
if (files != null) {
// encoded filename/checksum of files, write into _files
- try (BufferedWriter fileListWriter = writer(withinContext, qlPtn)) {
-for (String file : files) {
- writeFileEntry(qlMdTable.getDbName(), qlMdTable, file,
fileListWriter, withinContext);
-}
+ for (String file : files) {
Review comment:
tests for this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413976)
Time Spent: 0.5h (was: 20m)
> Avoid using _files for replication data copy during incremental run
> ---
>
> Key: HIVE-23020
> URL: https://issues.apache.org/jira/browse/HIVE-23020
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23020.01.patch, HIVE-23020.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23020) Avoid using _files for replication data copy during incremental run
[
https://issues.apache.org/jira/browse/HIVE-23020?focusedWorklogId=413974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413974
]
ASF GitHub Bot logged work on HIVE-23020:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:50
Start Date: 01/Apr/20 10:50
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #963: HIVE-23020 :
Avoid using _files for replication data copy during incr…
URL: https://github.com/apache/hive/pull/963#discussion_r401522879
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
##
@@ -505,9 +506,8 @@ private static ImportTableDesc
getBaseCreateTableDescFromTable(String dbName,
Task copyTask = null;
if (replicationSpec.isInReplicationScope()) {
- boolean isImport =
ReplicationSpec.Type.IMPORT.equals(replicationSpec.getReplSpecType());
copyTask = ReplCopyTask.getLoadCopyTask(replicationSpec, dataPath,
destPath, x.getConf(),
- isAutoPurge, needRecycle, copyToMigratedTxnTable, !isImport);
+ isAutoPurge, needRecycle, copyToMigratedTxnTable, false);
Review comment:
Why is this set to false? Do we even need this boolean if everyone is
setting it to false?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413974)
Time Spent: 20m (was: 10m)
> Avoid using _files for replication data copy during incremental run
> ---
>
> Key: HIVE-23020
> URL: https://issues.apache.org/jira/browse/HIVE-23020
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23020.01.patch, HIVE-23020.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.19.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23076) Add batching for openTxn
[
https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072629#comment-17072629
]
Hive QA commented on HIVE-23076:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998336/HIVE-23076.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18163 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998336 - PreCommit-HIVE-Build
> Add batching for openTxn
>
>
> Key: HIVE-23076
> URL: https://issues.apache.org/jira/browse/HIVE-23076
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23076.2.patch, HIVE-23076.3.patch,
> HIVE-23076.4.patch, HIVE-23076.5.patch, HIVE-23076.6.patch,
> HIVE-23076.7.patch, HIVE-23076.8.patch, HIVE-23076.9.patch, HIVE-23076.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413958
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401496558
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##
@@ -296,4 +300,17 @@ public static boolean includeAcidTableInDump(HiveConf
conf) {
public static boolean tableIncludedInReplScope(ReplScope replScope, String
tableName) {
return ((replScope == null) ||
replScope.tableIncludedInReplScope(tableName));
}
+
+ public static boolean dataCopyCompleted(Path toPath, HiveConf conf) throws
IOException {
+FileSystem dstFs = null;
+dstFs = toPath.getFileSystem(conf);
+if (dstFs.exists(new Path(toPath, ReplUtils.COPY_ACKNOWLEDGEMENT))) {
Review comment:
do return of condition directly rather than using if clause
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413958)
Time Spent: 0.5h (was: 20m)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413959
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401491670
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/ReplCopyWork.java
##
@@ -120,4 +122,12 @@ public boolean isNeedCheckDuplicateCopy() {
public void setCheckDuplicateCopy(boolean flag) {
checkDuplicateCopy = flag;
}
+
+ public boolean isCheckpointEnabled() {
+return checkpointEnabled;
+ }
+
+ public void setCheckpointEnabled(boolean checkpointEnabled) {
Review comment:
do initialization based on constructor rather than setters ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413959)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413960 ] ASF GitHub Bot logged work on HIVE-23039: - Author: ASF GitHub Bot Created on: 01/Apr/20 10:23 Start Date: 01/Apr/20 10:23 Worklog Time Spent: 10m Work Description: anishek commented on pull request #965: HIVE-23039 Checkpointing for repl dump bootstrap phase URL: https://github.com/apache/hive/pull/965#discussion_r401496301 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java ## @@ -98,6 +100,8 @@ public static final String DUMP_ACKNOWLEDGEMENT = "_finished_dump"; //Acknowledgement for repl load complete public static final String LOAD_ACKNOWLEDGEMENT = "_finished_load"; + //Acknowledgement for data copy complete. Used for checkpointing + public static final String COPY_ACKNOWLEDGEMENT = "_finished_copy"; Review comment: looks like we need a constants Class or a Enum class, the utils one seem to have a lot of magic variables. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 413960) Time Spent: 40m (was: 0.5h) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413961
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401496956
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##
@@ -296,4 +300,17 @@ public static boolean includeAcidTableInDump(HiveConf
conf) {
public static boolean tableIncludedInReplScope(ReplScope replScope, String
tableName) {
return ((replScope == null) ||
replScope.tableIncludedInReplScope(tableName));
}
+
+ public static boolean dataCopyCompleted(Path toPath, HiveConf conf) throws
IOException {
+FileSystem dstFs = null;
+dstFs = toPath.getFileSystem(conf);
+if (dstFs.exists(new Path(toPath, ReplUtils.COPY_ACKNOWLEDGEMENT))) {
+ return true;
+}
+return false;
+ }
+
+ public static void setDataCopyComplete(Path toPath, HiveConf conf) throws
SemanticException {
Review comment:
rename method to ackCopy() ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413961)
Time Spent: 50m (was: 40m)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413957
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401486847
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -906,6 +908,131 @@ public void replicationWithTableNameContainsKeywords()
throws Throwable {
.verifyReplTargetProperty(replicatedDbName);
}
+ @Test
+ public void testCheckPointing() throws Throwable {
+List withClauseOptions = externalTableBasePathWithClause();
+WarehouseInstance.Tuple bootstrapDump = primary.run("use " + primaryDbName)
+.run("CREATE TABLE t1(a string) STORED AS TEXTFILE")
+.run("CREATE EXTERNAL TABLE t2(a string) STORED AS TEXTFILE")
+.run("insert into t1 values (1)")
+.run("insert into t1 values (2)")
+.run("insert into t2 values (11)")
+.run("insert into t2 values (21)")
+.dump(primaryDbName, withClauseOptions);
+
+// verify that the external table info is written correctly for bootstrap
+assertExternalFileInfo(Arrays.asList("t2"), bootstrapDump.dumpLocation,
primaryDbName);
Review comment:
might be better to use another location here completely, sometimes since we
are writing to same dump location and also the db directory is there you might
have files in locations that are slight off and you wont realize it till you
run actual prod scenario
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413957)
Time Spent: 20m (was: 10m)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-5312) Let HiveServer2 run simultaneously in HTTP (over thrift) and Binary (normal thrift transport) mode
[ https://issues.apache.org/jira/browse/HIVE-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-5312: - Attachment: HIVE-5312.3.patch > Let HiveServer2 run simultaneously in HTTP (over thrift) and Binary (normal > thrift transport) mode > --- > > Key: HIVE-5312 > URL: https://issues.apache.org/jira/browse/HIVE-5312 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Vaibhav Gumashta >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-5132.branch-2.3.patch, HIVE-5312.1.patch, > HIVE-5312.2.patch, HIVE-5312.3.patch > > > [HIVE-4763|https://issues.apache.org/jira/browse/HIVE-4763] adds support for > HTTP transport over thrift. With that, HS2 can be configured to run either > using using HTTP or using normal thrift binary transport. Ideally HS2 should > be supporting both modes simultaneously and the client should be able to > specify the mode used in serving the request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
