[jira] [Resolved] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller resolved HIVE-22768. Release Note: The resulted list should be mutable, so the proposed solution won't work Resolution: Won't Fix > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22768: --- Status: Open (was: Patch Available) That was a misunderstanding. > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022734#comment-17022734
]
Hive QA commented on HIVE-22746:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991685/HIVE-22746.01.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20313/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20313/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20313/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991685 - PreCommit-HIVE-Build
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch,
> HIVE-22746.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22538:
--
Status: Patch Available (was: Open)
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch,
> HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22538:
--
Attachment: HIVE-22538.6.patch
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch,
> HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22538:
--
Status: Open (was: Patch Available)
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch,
> HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022725#comment-17022725
]
Hive QA commented on HIVE-22746:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 142 new + 1331 unchanged - 15
fixed = 1473 total (was 1346) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
1s{color} | {color:red} ql generated 1 new + 1530 unchanged - 2 fixed = 1531
total (was 1532) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
57s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of procCtx at line 1499 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.processGByExpr(Node,
Object) At TypeCheckProcFactory.java:1499 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.processGByExpr(Node,
Object) At TypeCheckProcFactory.java:[line 1492] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20313/dev-support/hive-personality.sh
|
| git revision | master / 4193b54 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus/new-findbugs-ql.html
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus/diff-javadoc-javadoc-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20313/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch,
> HIVE-22746.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid
[jira] [Commented] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022713#comment-17022713
]
Hive QA commented on HIVE-22767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991675/HIVE-22767.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17946 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20312/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20312/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20312/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991675 - PreCommit-HIVE-Build
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22769:
-
Summary: Incorrect query results and query failure during split generation
for compressed text files (was: Incorrect query results and query failure
during Split generation for compressed text files)
> Incorrect query results and query failure during split generation for
> compressed text files
> ---
>
> Key: HIVE-22769
> URL: https://issues.apache.org/jira/browse/HIVE-22769
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Priority: Critical
> Attachments: testcase1.csv.bz2, testcase2.csv.bz2
>
>
> Hive Query produces incorrect results when data is in text format and
> compressed and for certain data the query fails during split generation.
> This behavior is seen when skip.header.line.count and skip.footer.line.count
> are set for table.
> Case 1: Select count/aggregate query produces Incorrect row counts/displays
> all rows (when hive.fetch.task.conversion=none)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
> 3. Run count(*) on table.
> {code}
> > select * from testcase1;
> INFO : Compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string,
> comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.07 seconds
> INFO : Executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Completed executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.007 seconds
> INFO : OK
> +---+-+
> | testcase1.id | testcase1.name |
> +---+-+
> | 2 | 2019-12-31 |
> +---+-+
> 1 row selected (0.111 seconds)
> > select count(*) from testcase1
> INFO : Compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:bigint, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 0.073 seconds
> INFO : Executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Session is already open
> INFO : Dag name: select count(*) from testcase1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1579811438512_0046)
> .
> .
> .
> INFO : Completed executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 4.228 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 3|
> +--+
> 1 row selected (4.335 seconds)
> {code}
> Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
> java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
> The issue is only seen when there is a space in a field (eg:- "3,2019-12-31
> 01" second column has a space)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
> 3. Run count(*) on table.
> {code}
> 0: > select * from testcase2;
> INFO : Compiling
> command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
> select * from
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22769:
-
Description:
Hive Query produces incorrect results when data is in text format and
compressed and for certain data the query fails during split generation.
This behavior is seen when skip.header.line.count and skip.footer.line.count
are set for table.
Case 1: Select count/aggregate query produces Incorrect row counts/displays all
rows (when hive.fetch.task.conversion=none)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
3. Run count(*) on table.
{code}
> select * from testcase1;
INFO : Compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null),
FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Completed executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.007 seconds
INFO : OK
+---+-+
| testcase1.id | testcase1.name |
+---+-+
| 2 | 2019-12-31 |
+---+-+
1 row selected (0.111 seconds)
> select count(*) from testcase1
INFO : Compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
type:bigint, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 0.073 seconds
INFO : Executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Session is already open
INFO : Dag name: select count(*) from testcase1 (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1579811438512_0046)
.
.
.
INFO : Completed executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 4.228 seconds
INFO : OK
+--+
| _c0 |
+--+
| 3|
+--+
1 row selected (4.335 seconds)
{code}
Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01"
second column has a space)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
3. Run count(*) on table.
{code}
0: > select * from testcase2;
INFO : Compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null),
FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.075 seconds
INFO : Executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Completed executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.01 seconds
INFO : OK
+---+-+
| testcase2.id | testcase2.name |
+---+-+
| 2 | 2019-12-31 01 |
+---+-+
1 row selected (0.119 seconds)
> select count(*) from testcase2;
INFO : Compiling
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22769:
-
Description:
Hive Query produces incorrect results when data is in text format and
compressed and for certain data the query fails during split generation.
This behavior is seen when skip.header.line.count and skip.footer.line.count
are set for table.
Case 1: Select count/aggregate query produces Incorrect row counts/displays all
rows (when hive.fetch.task.conversion=none)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
3. Run count(*) on table.
{code}
> select * from testcase1;
INFO : Compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null),
FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Completed executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.007 seconds
INFO : OK
+---+-+
| testcase1.id | testcase1.name |
+---+-+
| 2 | 2019-12-31 |
+---+-+
1 row selected (0.111 seconds)
> select count(*) from testcase1
INFO : Compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
type:bigint, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 0.073 seconds
INFO : Executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Session is already open
INFO : Dag name: select count(*) from testcase1 (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1579811438512_0046)
.
.
.
INFO : Completed executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 4.228 seconds
INFO : OK
+--+
| _c0 |
+--+
| 3|
+--+
1 row selected (4.335 seconds)
{code}
Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01"
second column has a space)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
3. Run count(*) on table.
{code}
0: > select * from testcase2;
INFO : Compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null),
FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.075 seconds
INFO : Executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Completed executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.01 seconds
INFO : OK
+---+-+
| testcase2.id | testcase2.name |
+---+-+
| 2 | 2019-12-31 01 |
+---+-+
1 row selected (0.119 seconds)
> select count(*) from testcase2;
INFO : Compiling
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22769:
-
Description:
Hive Query produces incorrect results when data is in text format and
compressed and for certain data the query fails during split generation.
This behavior is seen when skip.header.line.count and skip.footer.line.count
are set for table.
Case 1: Select count/aggregate query produces Incorrect row counts/displays all
rows (when hive.fetch.task.conversion=none)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
3. Run count(*) on table.
{code}
> select * from testcase1;
INFO : Compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null),
FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Completed executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.007 seconds
INFO : OK
+---+-+
| testcase1.id | testcase1.name |
+---+-+
| 2 | 2019-12-31 |
+---+-+
1 row selected (0.111 seconds)
> select count(*) from testcase1
INFO : Compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
type:bigint, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 0.073 seconds
INFO : Executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Session is already open
INFO : Dag name: select count(*) from testcase1 (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1579811438512_0046)
.
.
.
INFO : Completed executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 4.228 seconds
INFO : OK
+--+
| _c0 |
+--+
| 3|
+--+
1 row selected (4.335 seconds)
{code}
Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01"
second column has a space)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
3. Run count(*) on table.
{code}
0: > select * from testcase2;
INFO : Compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null),
FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.075 seconds
INFO : Executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Completed executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.01 seconds
INFO : OK
+---+-+
| testcase2.id | testcase2.name |
+---+-+
| 2 | 2019-12-31 01 |
+---+-+
1 row selected (0.119 seconds)
{code}
> select count(*) from testcase2;
INFO : Compiling
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during Split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22769:
-
Description:
Hive Query produces incorrect results when data is in text format and
compressed and for certain data the query fails during split generation.
This behavior is seen when skip.header.line.count and skip.footer.line.count
are set for table.
Case 1: Select count/aggregate query produces Incorrect row counts/displays all
rows (when hive.fetch.task.conversion=none)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
3. Run count(**) on table.
{code}
> select * from testcase1;
INFO : Compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string, comment:null),
FieldSchema(name:testcase1.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
select * from testcase1
INFO : Completed executing
command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f); Time
taken: 0.007 seconds
INFO : OK
+---+-+
| testcase1.id | testcase1.name |
+---+-+
| 2 | 2019-12-31 |
+---+-+
1 row selected (0.111 seconds)
> select count(*) from testcase1
INFO : Compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
type:bigint, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 0.073 seconds
INFO : Executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
select count(*) from testcase1
INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Subscribed to counters: [] for queryId:
hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
INFO : Session is already open
INFO : Dag name: select count(*) from testcase1 (Stage-1)
INFO : Status: Running (Executing on YARN cluster with App id
application_1579811438512_0046)
.
.
.
INFO : Completed executing
command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7); Time
taken: 4.228 seconds
INFO : OK
+--+
| _c0 |
+--+
| 3|
+--+
1 row selected (4.335 seconds)
{code}
Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
The issue is only seen when there is a space in a field (eg:- "3,2019-12-31 01"
second column has a space)
Steps to reproduce:
1. Create table as below
{code}
CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
{code}
2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
3. Run count(*) on table.
{code}
0: > select * from testcase2;
INFO : Compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:testcase2.id, type:string, comment:null),
FieldSchema(name:testcase2.name, type:string, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.075 seconds
INFO : Executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
select * from testcase2
INFO : Completed executing
command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134); Time
taken: 0.01 seconds
INFO : OK
+---+-+
| testcase2.id | testcase2.name |
+---+-+
| 2 | 2019-12-31 01 |
+---+-+
1 row selected (0.119 seconds)
{code}
> select count(*) from testcase2;
INFO : Compiling
[jira] [Commented] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022689#comment-17022689
]
Hive QA commented on HIVE-22767:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} beeline in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} beeline: The patch generated 28 new + 36 unchanged - 0
fixed = 64 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20312/dev-support/hive-personality.sh
|
| git revision | master / 4193b54 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20312/yetus/diff-checkstyle-beeline.txt
|
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20312/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO :
[jira] [Commented] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[
https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022680#comment-17022680
]
Hive QA commented on HIVE-22768:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991673/HIVE-22768.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 17945 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=101)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20]
(batchId=102)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join21] (batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join22] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join28] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join29] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join31] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join3] (batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats2]
(batchId=100)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer5]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer7]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_join] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_rearrange]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join26] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join29] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=103)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_on_varchar]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_star] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[manyViewJoin]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin47] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_decimal_vectorized]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_distinct]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_mapjoin]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery2]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_subquery]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_join_union]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonblock_op_deduplicate]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[reduce_deduplicate_exclude_join]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin10]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin11]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin1]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin2]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin3]
(batchId=72)
[jira] [Commented] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[
https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022667#comment-17022667
]
Hive QA commented on HIVE-22768:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 10 new + 181 unchanged - 1
fixed = 191 total (was 182) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 15 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20311/dev-support/hive-personality.sh
|
| git revision | master / 4193b54 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20311/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove the usages of the Utilities.makeList method call
> ---
>
> Key: HIVE-22768
> URL: https://issues.apache.org/jira/browse/HIVE-22768
> Project: Hive
> Issue Type: Improvement
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Minor
> Attachments: HIVE-22768.1.patch
>
>
> The aforementioned
> org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non
> generic and most of its usages can be replaced by a simple
> Collections.singletonList(Object) call.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022660#comment-17022660 ] Eugene Chung commented on HIVE-22126: - Because of calcite dependency on guava, I think it should be included in hive-exec.jar too. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-10362) Support Type check/conversion in dynamic partition column
[
https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022659#comment-17022659
]
Hive QA commented on HIVE-10362:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991672/HIVE-10362.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 99 failed/errored test(s), 17945 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=303)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=306)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=306)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_part]
(batchId=306)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
(batchId=306)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_merge]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid2]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input13] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input45] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[inputwherefalse]
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_directory2]
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_static_ptn_into_bucketed_table]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge_empty] (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[null_column] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nullformatdir]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge10] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge11] (batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge1] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge2] (batchId=103)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_diff_fs]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_multi_insert]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_transform]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_join_partition_key]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subq] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_alias]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union] (batchId=5)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge10]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge_diff_fs]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schemeAuthority2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schemeAuthority]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_dir_distcp]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge11]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dml]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_insert_overwrite_local_directory_1]
(batchId=166)
[jira] [Commented] (HIVE-10362) Support Type check/conversion in dynamic partition column
[
https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022649#comment-17022649
]
Hive QA commented on HIVE-10362:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 353 unchanged - 1
fixed = 354 total (was 354) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20310/dev-support/hive-personality.sh
|
| git revision | master / 4193b54 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20310/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20310/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support Type check/conversion in dynamic partition column
> -
>
> Key: HIVE-10362
> URL: https://issues.apache.org/jira/browse/HIVE-10362
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, Types
>Affects Versions: 1.0.0, 3.0.0, 4.0.0
>Reporter: Chaoyu Tang
>Assignee: Karen Coppage
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-10362.01.patch
>
>
> There are quite a lot of issues associated with the non-noramlized or
> type-mismatched values for partition column. Hive has many ways to introduce
> such problematic data. HIVE-10307 mainly provides the support to type
> check/convert/normalize the partition column value in static partition
> specification. This JIRA tries to deal with the partition column type in
> dynamic partition insert. Currently any data can be inserted as a partition
> column value as long as it is quoted as a string. For example,
> create table dynparttypechecknum (key int, value string) partitioned by (part
> int);
> insert into dynparttypechecknum partition (part) select key, value, '1'
> from src limit 1;
> show partitions dynparttypechecknum;
> --
> part=1
> The partition column value is non-normalized int 1. It causes some
> unnecessary problems such as integer partition column JDO filter pushdown
> (see HIVE-6052) and others like HIVE-10210.
--
This message was sent by Atlassian Jira
[jira] [Commented] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[
https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022640#comment-17022640
]
Hive QA commented on HIVE-22726:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991666/HIVE-22726.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20309/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20309/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20309/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-24 01:40:18.344
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20309/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-24 01:40:18.346
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4193b54 HIVE-22751: Move locking in
HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper (Rajesh
Balamohan, reviewed by Anishek Agarwal)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4193b54 HIVE-22751: Move locking in
HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper (Rajesh
Balamohan, reviewed by Anishek Agarwal)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-24 01:40:19.523
+ rm -rf ../yetus_PreCommit-HIVE-Build-20309
+ mkdir ../yetus_PreCommit-HIVE-Build-20309
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20309
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20309/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperBatch.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneralComparator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java: does not
exist in index
Trying to apply the patch with -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with
conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with
conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58
Falling
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022639#comment-17022639
]
Hive QA commented on HIVE-20801:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991665/HIVE-20801.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17946 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20308/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20308/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20308/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991665 - PreCommit-HIVE-Build
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch,
> HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency despite the lock manager consuming resources
> centrally.
> Disabling this lock acquisition would improve the performance of non-ACID
> tables co-existing with a globally configured DbTxnManager implementation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022616#comment-17022616
]
Hive QA commented on HIVE-20801:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 20 new + 298 unchanged - 0
fixed = 318 total (was 298) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20308/dev-support/hive-personality.sh
|
| git revision | master / 4193b54 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20308/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20308/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Labels: Branch3Candidate, TODOC
> Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch,
> HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch
>
>
> Enabling ACIDv1 on a cluster produces a central locking bottleneck for all
> table types, which is not always the intention.
> The Hive locking for non-acid tables are advisory (i.e a client can
> write/read without locking), which means that the implementation does not
> offer strong consistency despite the lock manager consuming resources
> centrally.
> Disabling this lock acquisition would improve the performance of non-ACID
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022600#comment-17022600
]
Hive QA commented on HIVE-22538:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991657/HIVE-22538.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17946 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=180)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20307/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20307/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20307/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991657 - PreCommit-HIVE-Build
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022585#comment-17022585
]
Hive QA commented on HIVE-22538:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
47s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} ql: The patch generated 0 new + 499 unchanged - 4
fixed = 499 total (was 503) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 170
unchanged - 2 fixed = 170 total (was 172) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20307/dev-support/hive-personality.sh
|
| git revision | master / 4193b54 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20307/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
[jira] [Updated] (HIVE-22712) ReExec Driver execute submit the query in default queue irrespective of user defined queue
[ https://issues.apache.org/jira/browse/HIVE-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22712: -- Attachment: HIVE-22712.07.patch Status: Patch Available (was: Open) > ReExec Driver execute submit the query in default queue irrespective of user > defined queue > -- > > Key: HIVE-22712 > URL: https://issues.apache.org/jira/browse/HIVE-22712 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.1.2 > Environment: Hive-3 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22712.01.patch, HIVE-22712.02.patch, > HIVE-22712.03.patch, HIVE-22712.04.patch, HIVE-22712.05.patch, > HIVE-22712.06.patch, HIVE-22712.06.patch, HIVE-22712.07.patch, > HIVE-22712.patch > > > we unset the queue name intentionally in > TezSessionState#startSessionAndContainers, > as a result reexec create a new session in the default queue and create a > problem, its a cumbersome to add reexec.overlay.tez.queue.name at session > level. > I could not find a better way of setting the queue name (I am open for the > suggestion here) since it can create a conflict with the Global queue name > vs user-defined queue that's why setting while initialization of > ReExecutionOverlayPlugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22712) ReExec Driver execute submit the query in default queue irrespective of user defined queue
[ https://issues.apache.org/jira/browse/HIVE-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22712: -- Status: Open (was: Patch Available) > ReExec Driver execute submit the query in default queue irrespective of user > defined queue > -- > > Key: HIVE-22712 > URL: https://issues.apache.org/jira/browse/HIVE-22712 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.1.2 > Environment: Hive-3 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22712.01.patch, HIVE-22712.02.patch, > HIVE-22712.03.patch, HIVE-22712.04.patch, HIVE-22712.05.patch, > HIVE-22712.06.patch, HIVE-22712.06.patch, HIVE-22712.07.patch, > HIVE-22712.patch > > > we unset the queue name intentionally in > TezSessionState#startSessionAndContainers, > as a result reexec create a new session in the default queue and create a > problem, its a cumbersome to add reexec.overlay.tez.queue.name at session > level. > I could not find a better way of setting the queue name (I am open for the > suggestion here) since it can create a conflict with the Global queue name > vs user-defined queue that's why setting while initialization of > ReExecutionOverlayPlugin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[
https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022568#comment-17022568
]
Hive QA commented on HIVE-22518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991655/HIVE-22518.06.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17945 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=287)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20306/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20306/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20306/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991655 - PreCommit-HIVE-Build
> SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
>
>
> Key: HIVE-22518
> URL: https://issues.apache.org/jira/browse/HIVE-22518
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch,
> HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch,
> HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch,
> HIVE-22518.05.patch, HIVE-22518.06.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[
https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022563#comment-17022563
]
Hive QA commented on HIVE-22518:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
53s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20306/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql . itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20306/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
>
>
> Key: HIVE-22518
> URL: https://issues.apache.org/jira/browse/HIVE-22518
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch,
> HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch,
> HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch,
> HIVE-22518.05.patch, HIVE-22518.06.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[
https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-22751:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~anishek] for the review. Committed to master.
> Move locking in HiveServer2::isDeregisteredWithZooKeeper to
> ZooKeeperHiveHelper
> ---
>
> Key: HIVE-22751
> URL: https://issues.apache.org/jira/browse/HIVE-22751
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22751.1.patch
>
>
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597]
>
> When queries are run in beeline and closed, it causes unwanted delays in
> shutting down beeline. Here is the threaddump from server side, which shows
> HiveServer2 lock contention.
>
> It would be good to move synchronization to
> "zooKeeperHelper.isDeregisteredWithZooKeeper"
>
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on
> condition [0x7f78b968c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xac8d5ff0> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
> - locked <0xa9c5f2a8> (a java.lang.Object)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115)
> at
> org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763)
> at
> org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687)
> - locked <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016)
> at
> org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0
> tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry
> [0x7f7884edf000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600)
> - waiting to lock <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621)
> - locked <0xaa1970b0> (a
> org.apache.hive.service.cli.session.SessionManager)
> at
> org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:244)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseSession(ThriftCLIService.java:527)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1517)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1502)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at
>
[jira] [Commented] (HIVE-22744) TezTask for the vertex with more than one outedge should have proportional sort memory
[ https://issues.apache.org/jira/browse/HIVE-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022529#comment-17022529 ] Ashutosh Chauhan commented on HIVE-22744: - +1 > TezTask for the vertex with more than one outedge should have proportional > sort memory > -- > > Key: HIVE-22744 > URL: https://issues.apache.org/jira/browse/HIVE-22744 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22744.1.patch, HIVE-22744.2.patch > > > TezTask for the vertex with more than one outedge should have proportional > sort memory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022493#comment-17022493
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991653/HIVE-22729.08.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 135 failed/errored test(s), 17946 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=90)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindNextToClean
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindNextToCompact
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindNextToCompact2
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCompacted
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkFailed
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeFromLocalWorkers
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testShowCompactions
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCompactMajorWithPartition
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCompactMinorNoPartition
(batchId=314)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCompactWhenAlreadyCompacting
(batchId=314)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnMR (batchId=250)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=250)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnMR (batchId=250)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnTez (batchId=250)
org.apache.hadoop.hive.ql.TestTxnCommands.testVersioning (batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands2.testCompactWithDelete (batchId=340)
org.apache.hadoop.hive.ql.TestTxnCommands2.testEmptyInTblproperties
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCompactWithDelete
(batchId=354)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testEmptyInTblproperties
(batchId=354)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInitiatorWithMultipleFailedCompactions
(batchId=354)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=354)
org.apache.hadoop.hive.ql.TestTxnCommands3.testAcidMetaColumsDecode
(batchId=355)
org.apache.hadoop.hive.ql.TestTxnCommands3.testCompactionAbort (batchId=355)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOff
(batchId=355)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOn
(batchId=355)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testSnapshotIsolationWithAbortedTxnOnMmTable
(batchId=314)
org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testSnapshotIsolationWithAbortedTxnOnMmTable
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testVersioning
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenate (batchId=324)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenateMM (batchId=324)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenatePart (batchId=324)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=340)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNonAcidToAcidVectorzied
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult
(batchId=340)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testNonAcidToAcidVectorzied
(batchId=340)
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompaction
(batchId=325)
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorPartitionCompactionNoBase
(batchId=325)
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMajorTableCompaction
(batchId=325)
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorPartitionCompaction
(batchId=325)
org.apache.hadoop.hive.ql.txn.compactor.TestCleaner.cleanupAfterMinorTableCompaction
(batchId=325)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022470#comment-17022470
]
Hive QA commented on HIVE-22729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
49s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
20s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 3 new + 696 unchanged - 6 fixed = 699 total (was 702) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20305/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20305/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20305/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20305/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch
>
>
> We should provide a compaction failure reason as easily accessible as
>
[jira] [Updated] (HIVE-22393) HiveStreamingConnection: Exception in beginTransaction causes AbstractRecordWriter to throw NPE, covering up real exception
[
https://issues.apache.org/jira/browse/HIVE-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated HIVE-22393:
Attachment: HIVE-22393.1.patch
> HiveStreamingConnection: Exception in beginTransaction causes
> AbstractRecordWriter to throw NPE, covering up real exception
> ---
>
> Key: HIVE-22393
> URL: https://issues.apache.org/jira/browse/HIVE-22393
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.2
>Reporter: Alex Parrill
>Assignee: Matt Burgess
>Priority: Major
> Attachments: HIVE-22393.1.patch
>
>
> Getting an error in my code that does some basic stuff with
> {{HiveStreamingConnection}}:
> {code}
> 19/10/23 15:12:44 ERROR yarn_logger.Main: Thread worker-0 uncaught exception
> java.lang.RuntimeException: org.apache.hive.streaming.StreamingException:
> Unable to close
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:51)
> Caused by: org.apache.hive.streaming.StreamingException: Unable to close
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:973)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.markDead(HiveStreamingConnection.java:833)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:677)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:596)
> at
> org.apache.hive.streaming.HiveStreamingConnection.createNewTransactionBatch(HiveStreamingConnection.java:485)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginNextTransaction(HiveStreamingConnection.java:466)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginTransaction(HiveStreamingConnection.java:507)
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:49)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hive.streaming.AbstractRecordWriter.logStats(AbstractRecordWriter.java:547)
> at
> org.apache.hive.streaming.AbstractRecordWriter.close(AbstractRecordWriter.java:352)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.closeImpl(HiveStreamingConnection.java:979)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:970)
> ... 7 more
> {code}
> Digging through the stack trace... {{TransactionBatch}} will try to catch
> exception in its constructor, and calls the close method if an exception is
> thrown (which is definitely happening; that's line 677 in
> {{HiveStreamingConnection.java}}). This eventually calls
> {{AbstractRecordWriter::close}} which, calls {{logStats}}, which tries to use
> {{heapMemoryMonitor}}, which is null because presumably {{init}} had never
> been called on the writer yet.
> Easy fix would to just check in {{logStats}} if {{heapMemoryMonitor}} is
> null, and do the same thing if the method it calls returns null in it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22393) HiveStreamingConnection: Exception in beginTransaction causes AbstractRecordWriter to throw NPE, covering up real exception
[
https://issues.apache.org/jira/browse/HIVE-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated HIVE-22393:
Status: Patch Available (was: In Progress)
> HiveStreamingConnection: Exception in beginTransaction causes
> AbstractRecordWriter to throw NPE, covering up real exception
> ---
>
> Key: HIVE-22393
> URL: https://issues.apache.org/jira/browse/HIVE-22393
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.2
>Reporter: Alex Parrill
>Assignee: Matt Burgess
>Priority: Major
> Attachments: HIVE-22393.1.patch
>
>
> Getting an error in my code that does some basic stuff with
> {{HiveStreamingConnection}}:
> {code}
> 19/10/23 15:12:44 ERROR yarn_logger.Main: Thread worker-0 uncaught exception
> java.lang.RuntimeException: org.apache.hive.streaming.StreamingException:
> Unable to close
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:51)
> Caused by: org.apache.hive.streaming.StreamingException: Unable to close
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:973)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.markDead(HiveStreamingConnection.java:833)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:677)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:596)
> at
> org.apache.hive.streaming.HiveStreamingConnection.createNewTransactionBatch(HiveStreamingConnection.java:485)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginNextTransaction(HiveStreamingConnection.java:466)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginTransaction(HiveStreamingConnection.java:507)
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:49)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hive.streaming.AbstractRecordWriter.logStats(AbstractRecordWriter.java:547)
> at
> org.apache.hive.streaming.AbstractRecordWriter.close(AbstractRecordWriter.java:352)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.closeImpl(HiveStreamingConnection.java:979)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:970)
> ... 7 more
> {code}
> Digging through the stack trace... {{TransactionBatch}} will try to catch
> exception in its constructor, and calls the close method if an exception is
> thrown (which is definitely happening; that's line 677 in
> {{HiveStreamingConnection.java}}). This eventually calls
> {{AbstractRecordWriter::close}} which, calls {{logStats}}, which tries to use
> {{heapMemoryMonitor}}, which is null because presumably {{init}} had never
> been called on the writer yet.
> Easy fix would to just check in {{logStats}} if {{heapMemoryMonitor}} is
> null, and do the same thing if the method it calls returns null in it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22393) HiveStreamingConnection: Exception in beginTransaction causes AbstractRecordWriter to throw NPE, covering up real exception
[
https://issues.apache.org/jira/browse/HIVE-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned HIVE-22393:
---
Assignee: Matt Burgess
> HiveStreamingConnection: Exception in beginTransaction causes
> AbstractRecordWriter to throw NPE, covering up real exception
> ---
>
> Key: HIVE-22393
> URL: https://issues.apache.org/jira/browse/HIVE-22393
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.2
>Reporter: Alex Parrill
>Assignee: Matt Burgess
>Priority: Major
>
> Getting an error in my code that does some basic stuff with
> {{HiveStreamingConnection}}:
> {code}
> 19/10/23 15:12:44 ERROR yarn_logger.Main: Thread worker-0 uncaught exception
> java.lang.RuntimeException: org.apache.hive.streaming.StreamingException:
> Unable to close
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:51)
> Caused by: org.apache.hive.streaming.StreamingException: Unable to close
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:973)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.markDead(HiveStreamingConnection.java:833)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:677)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:596)
> at
> org.apache.hive.streaming.HiveStreamingConnection.createNewTransactionBatch(HiveStreamingConnection.java:485)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginNextTransaction(HiveStreamingConnection.java:466)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginTransaction(HiveStreamingConnection.java:507)
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:49)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hive.streaming.AbstractRecordWriter.logStats(AbstractRecordWriter.java:547)
> at
> org.apache.hive.streaming.AbstractRecordWriter.close(AbstractRecordWriter.java:352)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.closeImpl(HiveStreamingConnection.java:979)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:970)
> ... 7 more
> {code}
> Digging through the stack trace... {{TransactionBatch}} will try to catch
> exception in its constructor, and calls the close method if an exception is
> thrown (which is definitely happening; that's line 677 in
> {{HiveStreamingConnection.java}}). This eventually calls
> {{AbstractRecordWriter::close}} which, calls {{logStats}}, which tries to use
> {{heapMemoryMonitor}}, which is null because presumably {{init}} had never
> been called on the writer yet.
> Easy fix would to just check in {{logStats}} if {{heapMemoryMonitor}} is
> null, and do the same thing if the method it calls returns null in it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22393) HiveStreamingConnection: Exception in beginTransaction causes AbstractRecordWriter to throw NPE, covering up real exception
[
https://issues.apache.org/jira/browse/HIVE-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22393 started by Matt Burgess.
---
> HiveStreamingConnection: Exception in beginTransaction causes
> AbstractRecordWriter to throw NPE, covering up real exception
> ---
>
> Key: HIVE-22393
> URL: https://issues.apache.org/jira/browse/HIVE-22393
> Project: Hive
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 3.1.2
>Reporter: Alex Parrill
>Assignee: Matt Burgess
>Priority: Major
>
> Getting an error in my code that does some basic stuff with
> {{HiveStreamingConnection}}:
> {code}
> 19/10/23 15:12:44 ERROR yarn_logger.Main: Thread worker-0 uncaught exception
> java.lang.RuntimeException: org.apache.hive.streaming.StreamingException:
> Unable to close
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:51)
> Caused by: org.apache.hive.streaming.StreamingException: Unable to close
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:973)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.markDead(HiveStreamingConnection.java:833)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:677)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.(HiveStreamingConnection.java:596)
> at
> org.apache.hive.streaming.HiveStreamingConnection.createNewTransactionBatch(HiveStreamingConnection.java:485)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginNextTransaction(HiveStreamingConnection.java:466)
> at
> org.apache.hive.streaming.HiveStreamingConnection.beginTransaction(HiveStreamingConnection.java:507)
> at com.datto.yarn_logger.WorkerThread.run(WorkerThread.java:49)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hive.streaming.AbstractRecordWriter.logStats(AbstractRecordWriter.java:547)
> at
> org.apache.hive.streaming.AbstractRecordWriter.close(AbstractRecordWriter.java:352)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.closeImpl(HiveStreamingConnection.java:979)
> at
> org.apache.hive.streaming.HiveStreamingConnection$TransactionBatch.close(HiveStreamingConnection.java:970)
> ... 7 more
> {code}
> Digging through the stack trace... {{TransactionBatch}} will try to catch
> exception in its constructor, and calls the close method if an exception is
> thrown (which is definitely happening; that's line 677 in
> {{HiveStreamingConnection.java}}). This eventually calls
> {{AbstractRecordWriter::close}} which, calls {{logStats}}, which tries to use
> {{heapMemoryMonitor}}, which is null because presumably {{init}} had never
> been called on the writer yet.
> Easy fix would to just check in {{logStats}} if {{heapMemoryMonitor}} is
> null, and do the same thing if the method it calls returns null in it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022420#comment-17022420
]
Hive QA commented on HIVE-22736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991649/HIVE-22736.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20304/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20304/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20304/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12991649/HIVE-22736.patch was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991649 - PreCommit-HIVE-Build
> Support replication across multiple encryption zones
>
>
> Key: HIVE-22736
> URL: https://issues.apache.org/jira/browse/HIVE-22736
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch,
> HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022403#comment-17022403
]
Hive QA commented on HIVE-22126:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20303/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20303/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Eugene Chung
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Status: Patch Available (was: Open)
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch,
> HIVE-22366.6.patch, HIVE-22366.7.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22744) TezTask for the vertex with more than one outedge should have proportional sort memory
[ https://issues.apache.org/jira/browse/HIVE-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22744: Attachment: HIVE-22744.2.patch Status: Patch Available (was: Open) > TezTask for the vertex with more than one outedge should have proportional > sort memory > -- > > Key: HIVE-22744 > URL: https://issues.apache.org/jira/browse/HIVE-22744 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22744.1.patch, HIVE-22744.2.patch > > > TezTask for the vertex with more than one outedge should have proportional > sort memory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22744) TezTask for the vertex with more than one outedge should have proportional sort memory
[ https://issues.apache.org/jira/browse/HIVE-22744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22744: Status: Open (was: Patch Available) > TezTask for the vertex with more than one outedge should have proportional > sort memory > -- > > Key: HIVE-22744 > URL: https://issues.apache.org/jira/browse/HIVE-22744 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22744.1.patch, HIVE-22744.2.patch > > > TezTask for the vertex with more than one outedge should have proportional > sort memory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore
[ https://issues.apache.org/jira/browse/HIVE-22756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22756: Attachment: HIVE-22756.2.patch Status: Patch Available (was: Open) > Drop XAttrs options for distcp when dest is a blobstore > --- > > Key: HIVE-22756 > URL: https://issues.apache.org/jira/browse/HIVE-22756 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22756.1.patch, HIVE-22756.2.patch > > > Drop XAttrs options for distcp when dest is a blobstore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore
[ https://issues.apache.org/jira/browse/HIVE-22756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22756: Status: Open (was: Patch Available) > Drop XAttrs options for distcp when dest is a blobstore > --- > > Key: HIVE-22756 > URL: https://issues.apache.org/jira/browse/HIVE-22756 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22756.1.patch > > > Drop XAttrs options for distcp when dest is a blobstore -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[
https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022374#comment-17022374
]
Hive QA commented on HIVE-22751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991397/HIVE-22751.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20302/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20302/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20302/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991397 - PreCommit-HIVE-Build
> Move locking in HiveServer2::isDeregisteredWithZooKeeper to
> ZooKeeperHiveHelper
> ---
>
> Key: HIVE-22751
> URL: https://issues.apache.org/jira/browse/HIVE-22751
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22751.1.patch
>
>
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/session/SessionManager.java#L597]
>
> When queries are run in beeline and closed, it causes unwanted delays in
> shutting down beeline. Here is the threaddump from server side, which shows
> HiveServer2 lock contention.
>
> It would be good to move synchronization to
> "zooKeeperHelper.isDeregisteredWithZooKeeper"
>
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f78b0078800 nid=0x2d1c waiting on
> condition [0x7f78b968c000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xac8d5ff0> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:187)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123)
> - locked <0xa9c5f2a8> (a java.lang.Object)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115)
> at
> org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:790)
> at
> org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:763)
> at
> org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:687)
> - locked <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1016)
> at
> org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:137)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1294)
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> "HiveServer2-HttpHandler-Pool: Thread-50" #50 prio=5 os_prio=0
> tid=0x7f78b3e60800 nid=0x2fa7 waiting for monitor entry
> [0x7f7884edf000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hive.service.server.HiveServer2.isDeregisteredWithZooKeeper(HiveServer2.java:600)
> - waiting to lock <0xa99bd568> (a
> org.apache.hive.service.server.HiveServer2)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSessionInternal(SessionManager.java:631)
> at
> org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:621)
> - locked <0xaa1970b0> (a
>
[jira] [Commented] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022349#comment-17022349
]
Vineet Garg commented on HIVE-22366:
https://github.com/apache/hive/pull/888
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch,
> HIVE-22366.6.patch, HIVE-22366.7.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Status: Open (was: Patch Available)
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch,
> HIVE-22366.6.patch, HIVE-22366.7.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?focusedWorklogId=376411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376411
]
ASF GitHub Bot logged work on HIVE-22366:
-
Author: ASF GitHub Bot
Created on: 23/Jan/20 18:22
Start Date: 23/Jan/20 18:22
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #888:
HIVE-22366: Multiple metastore calls for same table and constraints d…
URL: https://github.com/apache/hive/pull/888
…uring planning
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376411)
Remaining Estimate: 0h
Time Spent: 10m
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch,
> HIVE-22366.6.patch, HIVE-22366.7.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Attachment: HIVE-22366.7.patch
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch,
> HIVE-22366.6.patch, HIVE-22366.7.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-22366:
--
Labels: pull-request-available (was: )
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch,
> HIVE-22366.6.patch, HIVE-22366.7.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22751) Move locking in HiveServer2::isDeregisteredWithZooKeeper to ZooKeeperHiveHelper
[
https://issues.apache.org/jira/browse/HIVE-22751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022314#comment-17022314
]
Hive QA commented on HIVE-22751:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
53s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
59s{color} | {color:red} service generated 1 new + 50 unchanged - 0 fixed = 51
total (was 50) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:service |
| | Inconsistent synchronization of
org.apache.hive.service.server.HiveServer2.zooKeeperHelper; locked 62% of time
Unsynchronized access at HiveServer2.java:62% of time Unsynchronized access at
HiveServer2.java:[line 620] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20302/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20302/yetus/new-findbugs-service.html
|
| modules | C: standalone-metastore/metastore-common service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20302/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move locking in HiveServer2::isDeregisteredWithZooKeeper to
> ZooKeeperHiveHelper
> ---
>
> Key: HIVE-22751
> URL: https://issues.apache.org/jira/browse/HIVE-22751
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22751.1.patch
>
>
> [https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/server/HiveServer2.java#L620]
>
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022306#comment-17022306
]
Jesus Camacho Rodriguez commented on HIVE-22746:
[~vgarg], [~mgergely], can any of you review this patch?
https://github.com/apache/hive/pull/887
Cc [~scarlin]
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch,
> HIVE-22746.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-22746:
--
Labels: pull-request-available (was: )
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch,
> HIVE-22746.patch
>
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22746:
---
Attachment: HIVE-22746.01.patch
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch,
> HIVE-22746.patch
>
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?focusedWorklogId=376377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376377
]
ASF GitHub Bot logged work on HIVE-22746:
-
Author: ASF GitHub Bot
Created on: 23/Jan/20 17:40
Start Date: 23/Jan/20 17:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #887: HIVE-22746
URL: https://github.com/apache/hive/pull/887
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376377)
Remaining Estimate: 0h
Time Spent: 10m
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22746.01.patch, HIVE-22746.01.patch,
> HIVE-22746.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022287#comment-17022287
]
Hive QA commented on HIVE-21487:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991644/HIVE-21847.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20301/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20301/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20301/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991644 - PreCommit-HIVE-Build
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022266#comment-17022266 ] Naveen Gangam commented on HIVE-22679: -- [~belugabehr] Change looks good to me. So +1 for me. > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22767:
-
Status: Patch Available (was: Open)
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus updated HIVE-22767:
-
Attachment: HIVE-22767.01.patch
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-22767.01.patch
>
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22768: --- Assignee: Ivan Suller Status: Patch Available (was: Open) > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22768) Remove the usages of the Utilities.makeList method call
[ https://issues.apache.org/jira/browse/HIVE-22768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Suller updated HIVE-22768: --- Attachment: HIVE-22768.1.patch > Remove the usages of the Utilities.makeList method call > --- > > Key: HIVE-22768 > URL: https://issues.apache.org/jira/browse/HIVE-22768 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-22768.1.patch > > > The aforementioned > org.apache.hadoop.hive.ql.exec.Utilities.makeList(Object...) method is non > generic and most of its usages can be replaced by a simple > Collections.singletonList(Object) call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[
https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022236#comment-17022236
]
Hive QA commented on HIVE-21487:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20301/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20301/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
>
>
> Key: HIVE-21487
> URL: https://issues.apache.org/jira/browse/HIVE-21487
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Todd Lipcon
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch
>
>
> Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant
> stream of queries of the form:
> {code}
> select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE =
> 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION =
> 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc;
> {code}
> but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting
> in a full table scan over 115k rows, which takes around 100ms.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-10362) Support Type check/conversion in dynamic partition column
[ https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-10362: - Attachment: HIVE-10362.01.patch Fix Version/s: 4.0.0 Affects Version/s: 4.0.0 3.0.0 Status: Patch Available (was: Open) > Support Type check/conversion in dynamic partition column > - > > Key: HIVE-10362 > URL: https://issues.apache.org/jira/browse/HIVE-10362 > Project: Hive > Issue Type: Improvement > Components: Query Processor, Types >Affects Versions: 3.0.0, 1.0.0, 4.0.0 >Reporter: Chaoyu Tang >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-10362.01.patch > > > There are quite a lot of issues associated with the non-noramlized or > type-mismatched values for partition column. Hive has many ways to introduce > such problematic data. HIVE-10307 mainly provides the support to type > check/convert/normalize the partition column value in static partition > specification. This JIRA tries to deal with the partition column type in > dynamic partition insert. Currently any data can be inserted as a partition > column value as long as it is quoted as a string. For example, > create table dynparttypechecknum (key int, value string) partitioned by (part > int); > insert into dynparttypechecknum partition (part) select key, value, '1' > from src limit 1; > show partitions dynparttypechecknum; > -- > part=1 > The partition column value is non-normalized int 1. It causes some > unnecessary problems such as integer partition column JDO filter pushdown > (see HIVE-6052) and others like HIVE-10210. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-10362) Support Type check/conversion in dynamic partition column
[ https://issues.apache.org/jira/browse/HIVE-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-10362: Assignee: Karen Coppage (was: Chaoyu Tang) > Support Type check/conversion in dynamic partition column > - > > Key: HIVE-10362 > URL: https://issues.apache.org/jira/browse/HIVE-10362 > Project: Hive > Issue Type: Improvement > Components: Query Processor, Types >Affects Versions: 1.0.0 >Reporter: Chaoyu Tang >Assignee: Karen Coppage >Priority: Major > > There are quite a lot of issues associated with the non-noramlized or > type-mismatched values for partition column. Hive has many ways to introduce > such problematic data. HIVE-10307 mainly provides the support to type > check/convert/normalize the partition column value in static partition > specification. This JIRA tries to deal with the partition column type in > dynamic partition insert. Currently any data can be inserted as a partition > column value as long as it is quoted as a string. For example, > create table dynparttypechecknum (key int, value string) partitioned by (part > int); > insert into dynparttypechecknum partition (part) select key, value, '1' > from src limit 1; > show partitions dynparttypechecknum; > -- > part=1 > The partition column value is non-normalized int 1. It causes some > unnecessary problems such as integer partition column JDO filter pushdown > (see HIVE-6052) and others like HIVE-10210. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022200#comment-17022200
]
Hive QA commented on HIVE-22736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991649/HIVE-22736.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 17958 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_ddl1]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=175)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testPartitionOpsForUpdateUsingEvents
(batchId=249)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatistics
(batchId=249)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableColumnStatisticsTxnTable
(batchId=249)
org.apache.hadoop.hive.metastore.cache.TestCachedStoreUpdateUsingEvents.testTableOpsForUpdateUsingEvents
(batchId=249)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing
(batchId=275)
org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMajorPartitionCompaction
(batchId=279)
org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMajorTableCompaction
(batchId=279)
org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMinorPartitionCompaction
(batchId=279)
org.apache.hadoop.hive.ql.txn.compactor.TestCleanerWithReplication.cleanupAfterMinorTableCompaction
(batchId=279)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.insertPartition
(batchId=297)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.insertTable
(batchId=297)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlCTAS
(batchId=297)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertPartition
(batchId=297)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.sqlInsertTable
(batchId=297)
org.apache.hive.jdbc.TestJdbcDriver2.testGetQueryLogForReplCommands
(batchId=289)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testReplDumpResultSet (batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20300/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20300/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20300/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991649 - PreCommit-HIVE-Build
> Support replication across multiple encryption zones
>
>
> Key: HIVE-22736
> URL: https://issues.apache.org/jira/browse/HIVE-22736
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch,
> HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022189#comment-17022189 ] David Mollitor commented on HIVE-22679: --- [~ngangam] Can you please take a look at this one? > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22679: -- Attachment: (was: HIVE-22679.1.patch) > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22679) Replace Base64 in metastore-common Package
[ https://issues.apache.org/jira/browse/HIVE-22679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22679: -- Attachment: (was: HIVE-22679.1.patch) > Replace Base64 in metastore-common Package > -- > > Key: HIVE-22679 > URL: https://issues.apache.org/jira/browse/HIVE-22679 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22679.1.patch, HIVE-22679.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022182#comment-17022182
]
Hive QA commented on HIVE-22736:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
17s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
23s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/common: The patch generated 1 new + 93 unchanged
- 1 fixed = 94 total (was 94) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 3 new + 98 unchanged - 0 fixed = 101 total (was 98) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Assigned] (HIVE-22767) beeline doesn't parse semicolons in comments properly
[
https://issues.apache.org/jira/browse/HIVE-22767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Matyus reassigned HIVE-22767:
Assignee: Zoltan Matyus
> beeline doesn't parse semicolons in comments properly
> -
>
> Key: HIVE-22767
> URL: https://issues.apache.org/jira/browse/HIVE-22767
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Matyus
>Assignee: Zoltan Matyus
>Priority: Major
>
> HIVE-12646 fixed the handling of semicolons in quoted strings, but leaves the
> problem of semicolons in comments. E.g. with beeline connected to any
> database...
> this works: {code:sql}select 1; select /* */ 2; select /* */ 3;{code}
> this doesn't work: {code:sql}select 1; select /* ; */ 2; select /* ; */
> 3;{code}
> This has been fixed and reintroduced before (possibly multiple times).
> Ideally, there should be a single utility method somewhere to separate
> comments, strings and commands -- with the proper testing in place (q files).
> However, I'm trying to make this fix back-portable, so a light touch is
> needed. I'm focusing on beeline for now, and only writing (very thorough)
> unit tests, as I cannot exclude any new q files from TestCliDriver (which
> would break, since it's using a different parsing method).
>
> P.S. excerpt of the error message:
> {noformat}
> 0: jdbc:hive2://...> select 1; select /* ; */ 2; select /* ; */ 3;
> INFO : Compiling command(queryId=...): select 1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=...); Time taken: 0.38 seconds
> INFO : Executing command(queryId=...): select 1
> INFO : Completed executing command(queryId=...); Time taken: 0.004 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 1|
> +--+
> 1 row selected (2.007 seconds)
> INFO : Compiling command(queryId=...): select /*
> ERROR : FAILED: ParseException line 1:9 cannot recognize input near ''
> '' '' in select clause
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:9 cannot recognize
> input near '' '' '' in select clause
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:598)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1505)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1452)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1447)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at ...
> {noformat}
>
> Similarly, the following query also fails:
> {code:sql}select /* ' */ 1; select /* ' */ 2;{code}
> I suspect line comments are also not handled properly but I cannot reproduce
> this in interactive beeline...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376259
]
ASF GitHub Bot logged work on HIVE-22736:
-
Author: ASF GitHub Bot
Created on: 23/Jan/20 14:34
Start Date: 23/Jan/20 14:34
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #883: HIVE-22736
Support multiple encryption zones in replication
URL: https://github.com/apache/hive/pull/883#discussion_r370152190
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
##
@@ -213,9 +227,18 @@ public int recycle(Path path, RecycleType type, boolean
ifPurge) throws IOExcept
switch (type) {
case MOVE: {
LOG.info("Moving {} to {}", path.toString(), cmPath.toString());
-
// Rename fails if the file with same name already exist.
- success = fs.rename(path, cmPath);
+ Retry retriable = new Retry(IOException.class) {
+@Override
+public Boolean execute() throws IOException {
+ return fs.rename(path, cmPath);
+}
+ };
+ try {
+success = retriable.run();
+ } catch (Exception e) {
+throw new
MetaException(org.apache.hadoop.util.StringUtils.stringifyException(e));
Review comment:
Yes in this case it is just an IOException. But wanted to keep the Retry
interface generic. So added Exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376259)
Time Spent: 1h 20m (was: 1h 10m)
> Support replication across multiple encryption zones
>
>
> Key: HIVE-22736
> URL: https://issues.apache.org/jira/browse/HIVE-22736
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch,
> HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?focusedWorklogId=376258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-376258
]
ASF GitHub Bot logged work on HIVE-22736:
-
Author: ASF GitHub Bot
Created on: 23/Jan/20 14:32
Start Date: 23/Jan/20 14:32
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #883: HIVE-22736
Support multiple encryption zones in replication
URL: https://github.com/apache/hive/pull/883#discussion_r370151392
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java
##
@@ -138,15 +149,18 @@ private ReplChangeManager(Configuration conf) throws
MetaException {
if (!inited) {
if (MetastoreConf.getBoolVar(conf, ConfVars.REPLCMENABLED)) {
ReplChangeManager.enabled = true;
- ReplChangeManager.cmroot = new Path(MetastoreConf.getVar(conf,
ConfVars.REPLCMDIR));
ReplChangeManager.conf = conf;
+ //Create default cm root
+ Path cmroot = new Path(MetastoreConf.getVar(conf,
ConfVars.REPLCMDIR));
FileSystem cmFs = cmroot.getFileSystem(conf);
// Create cmroot with permission 700 if not exist
if (!cmFs.exists(cmroot)) {
cmFs.mkdirs(cmroot);
cmFs.setPermission(cmroot, new FsPermission("700"));
}
+ cmRootMapping.put(NO_ENCRYPTION, cmroot);
Review comment:
This is for REPLCMDIR. This will always be unencrypted. Encryption cm
directory we have separated. If there is any unencrypted path for which recycle
is called, we will use this. Else we will create a cm for that zone
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 376258)
Time Spent: 1h 10m (was: 1h)
> Support replication across multiple encryption zones
>
>
> Key: HIVE-22736
> URL: https://issues.apache.org/jira/browse/HIVE-22736
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch,
> HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: (was: HIVE-22726.3.patch) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Open (was: Patch Available) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: HIVE-22726.3.patch > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Patch Available (was: Open) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved HIVE-22766. - Resolution: Won't Fix > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-22766: --- Assignee: (was: László Bodor) > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022142#comment-17022142 ] László Bodor commented on HIVE-22766: - Sorry, I've just realized that upstream it's already done by HIVE-21440, closing this as invalid. > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[
https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022140#comment-17022140
]
Hive QA commented on HIVE-22764:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991626/HIVE-22764.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17946 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=291)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20299/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20299/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20299/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991626 - PreCommit-HIVE-Build
> Create new command for "optimize" compaction and have basic implementation.
> ---
>
> Key: HIVE-22764
> URL: https://issues.apache.org/jira/browse/HIVE-22764
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aditya Shah
>Assignee: Aditya Shah
>Priority: Major
> Attachments: HIVE-22764.patch
>
>
> Created new blocking compaction (added compaction type "optimize") by adding
> a lock request on the compaction's transaction. It works mostly like
> mmMajorCompaction and writes files w/o row_IDs. I have added an additional
> table property to provide optimize columns that is used by the compactor to
> cluster the data by.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22766: Description: Kryo 4 FieldSerializer is more flexible in a way that allows us to change the serialization/deserialization logic to be changed more easily. From performance point of view, I'm not sure, but I don't think we need serious benefit there. Kryo 4.0.0 release notes: https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading was: Kryo 4 FieldSerializer is more flexible in a way that it allows us to change the serialization/deserialization logic to be changed more easily. From performance point of view, I'm not sure, but I don't think we need serious benefit there. > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that allows us to change the > serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. > Kryo 4.0.0 release notes: > https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 > https://github.com/EsotericSoftware/kryo#kryo-versioning-and-upgrading -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22766: Description: Kryo 4 FieldSerializer is more flexible in a way that it allows us to change the serialization/deserialization logic to be changed more easily. From performance point of view, I'm not sure, but I don't think we need serious benefit there. > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > Kryo 4 FieldSerializer is more flexible in a way that it allows us to change > the serialization/deserialization logic to be changed more easily. From > performance point of view, I'm not sure, but I don't think we need serious > benefit there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22766) Upgrade to Kryo4
[ https://issues.apache.org/jira/browse/HIVE-22766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-22766: --- Assignee: László Bodor > Upgrade to Kryo4 > > > Key: HIVE-22766 > URL: https://issues.apache.org/jira/browse/HIVE-22766 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022130#comment-17022130 ] Denys Kuzmenko commented on HIVE-20801: --- rebased > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-20801: - Assignee: Gopal Vijayaraghavan (was: Denys Kuzmenko) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Gopal Vijayaraghavan >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-20801: -- Attachment: HIVE-20801.4.patch > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-20801: - Assignee: Denys Kuzmenko (was: Gopal Vijayaraghavan) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch, HIVE-20801.4.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[
https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022100#comment-17022100
]
Hive QA commented on HIVE-22764:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
33s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 10 new + 705 unchanged - 0 fixed = 715 total (was 705) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 7 new + 46 unchanged - 2 fixed
= 53 total (was 48) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 11
unchanged - 0 fixed = 15 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20299/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20299/yetus.txt |
| Powered by | Apache Yetus
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22538:
--
Status: Open (was: Patch Available)
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22538:
--
Attachment: HIVE-22538.6.patch
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22538:
--
Status: Patch Available (was: Open)
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.6.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022052#comment-17022052
]
Hive QA commented on HIVE-22538:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991623/HIVE-22538.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17946 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[clusterctas] (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=91)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20298/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20298/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20298/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991623 - PreCommit-HIVE-Build
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property might be overriden to 1, which can
> lead to merging final aggregation into a single stage (hence leading to
> performance degradation). For instance, when autogather column stats is
> enabled, this can happen for the following query:
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> EXPLAIN
> CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
> SELECT * FROM SRC x CLUSTER BY x.key;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022033#comment-17022033 ] Zoltan Haindrich commented on HIVE-22518: - we were talking about this with [~mgergely], and he suggested to not set user.name in the qoption; but try to also migrate the other existing tests to use this option to set up authorization...for the latter; I'll open a followup - as that will be "only" a "bulk" change of q files... > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch, > HIVE-22518.05.patch, HIVE-22518.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22518: Attachment: HIVE-22518.06.patch > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch, > HIVE-22518.05.patch, HIVE-22518.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.08.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22765) Quote table names in CompactionTxnHandler
[
https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter reassigned HIVE-22765:
> Quote table names in CompactionTxnHandler
> -
>
> Key: HIVE-22765
> URL: https://issues.apache.org/jira/browse/HIVE-22765
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Laszlo Pinter
>Assignee: Zoltan Chovan
>Priority: Major
>
> When I run a metastore backed up by a postgres db, I get the following
> exception:
> {code:java}
> MetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not existMetaException(message:Unable to connect to transaction database
> org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does
> not exist Position: 83 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at
> org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code}
> This is due to missing quotes around table names.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22538) RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer
[
https://issues.apache.org/jira/browse/HIVE-22538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021993#comment-17021993
]
Hive QA commented on HIVE-22538:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 6 new + 500 unchanged - 3
fixed = 506 total (was 503) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 170
unchanged - 2 fixed = 172 total (was 172) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20298/dev-support/hive-personality.sh
|
| git revision | master / 6357dbc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20298/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20298/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20298/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> RS deduplication does not always enforce
> hive.optimize.reducededuplication.min.reducer
> --
>
> Key: HIVE-22538
> URL: https://issues.apache.org/jira/browse/HIVE-22538
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22538.2.patch, HIVE-22538.3.patch,
> HIVE-22538.4.patch, HIVE-22538.5.patch, HIVE-22538.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For transactional tables, that property
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch, > HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021977#comment-17021977
]
Hive QA commented on HIVE-22736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991629/HIVE-22736.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20297/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20297/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12991629/HIVE-22736.patch was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991629 - PreCommit-HIVE-Build
> Support replication across multiple encryption zones
>
>
> Key: HIVE-22736
> URL: https://issues.apache.org/jira/browse/HIVE-22736
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch,
> HIVE-22736.patch, HIVE-22736.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021976#comment-17021976
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991615/HIVE-22729.07.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20296/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20296/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-23 11:19:03.943
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20296/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 11:19:03.945
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 6357dbc HIVE-22627: Add schema changes introduced in HIVE-21443
to the schema upgrade scripts (Zoltan Chovan via Peter Vary)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 6357dbc HIVE-22627: Add schema changes introduced in HIVE-21443
to the schema upgrade scripts (Zoltan Chovan via Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 11:19:04.664
+ rm -rf ../yetus_PreCommit-HIVE-Build-20296
+ mkdir ../yetus_PreCommit-HIVE-Build-20296
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20296
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20296/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java: does
not exist in index
error:
a/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java:
does not exist in index
error: a/ql/src/test/results/clientpositive/dbtxnmgr_showlocks.q.out: does not
exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
does not exist in index
error:
