[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219178&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219178
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 06:58
Start Date: 27/Mar/19 06:58
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269425412
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
##
@@ -1247,17 +1244,37 @@ private static void createReplImportTasks(
} else if (!replicationSpec.isMetadataOnly()
&& !shouldSkipDataCopyInReplScope(tblDesc, replicationSpec)) {
x.getLOG().debug("adding dependent CopyWork/MoveWork for table");
-t.addDependentTask(loadTable(fromURI, table,
replicationSpec.isReplace(),
-new Path(tblDesc.getLocation()), replicationSpec, x, writeId,
stmtId));
+dependentTasks = new ArrayList<>(1);
+dependentTasks.add(loadTable(fromURI, table,
replicationSpec.isReplace(),
+ new Path(tblDesc.getLocation()),
replicationSpec,
+ x, writeId, stmtId));
}
- if (dropTblTask != null) {
-// Drop first and then create
-dropTblTask.addDependentTask(t);
-x.getTasks().add(dropTblTask);
+ // During replication, by the time we reply a commit transaction event,
the table should
+ // have been already created when replaying previous events. So no need
to create table
+ // again. For some reason we need create table task for partitioned
table though.
Review comment:
Corrected. The partition case is already fixed, but the comment wasn't
corrected.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219178)
Time Spent: 3h (was: 2h 50m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21402) Compaction state remains 'working' when major compaction fails
[
https://issues.apache.org/jira/browse/HIVE-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802498#comment-16802498
]
Vaibhav Gumashta commented on HIVE-21402:
-
[~pvary] How about we catch the throwable, do the clean up and then throw it
again?
> Compaction state remains 'working' when major compaction fails
> --
>
> Key: HIVE-21402
> URL: https://issues.apache.org/jira/browse/HIVE-21402
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-21402.patch
>
>
> When calcite is not on the HMS classpath, and query based compaction is
> enabled then the compaction fails with NoClassDefFound error. Since the catch
> block only catches Exceptions the following code block is not executed:
> {code:java}
> } catch (Exception e) {
> LOG.error("Caught exception while trying to compact " + ci +
> ". Marking failed to avoid repeated failures, " +
> StringUtils.stringifyException(e));
> msc.markFailed(CompactionInfo.compactionInfoToStruct(ci));
> msc.abortTxns(Collections.singletonList(compactorTxnId));
> }
> {code}
> So the compaction is not set to failed.
> Would be better to catch Throwable instead of Exception
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Status: Open (was: Patch Available) > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Attachment: HIVE-21516.02.patch > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Status: Patch Available (was: Open) > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch, HIVE-21516.02.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219177&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219177
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 06:52
Start Date: 27/Mar/19 06:52
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269424107
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -828,6 +828,8 @@ public void alterPartitions(String tblName,
List newParts,
new ArrayList();
try {
AcidUtils.TableSnapshot tableSnapshot = null;
+ // TODO: In case of replication use the writeId and valid write id list
constructed for
Review comment:
I have addressed this comment and removed it as well. But didn't commit the
change and thus wasn't part of the PR. I have updated PR. This TODO is no more
there. Sorry.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219177)
Time Spent: 2h 50m (was: 2h 40m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=219176&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-219176
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 27/Mar/19 06:51
Start Date: 27/Mar/19 06:51
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #579:
HIVE-21109 : Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269423978
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
##
@@ -2689,7 +2689,19 @@ private int alterTable(Hive db, AlterTableDesc
alterTbl) throws HiveException {
} else {
// Note: this is necessary for UPDATE_STATISTICS command, that
operates via ADDPROPS (why?).
// For any other updates, we don't want to do txn check on
partitions when altering table.
-boolean isTxn = alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS;
+boolean isTxn = false;
+if (alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS) {
+ // ADDPROPS is used to add repl.last.id during replication. That's
not a transactional
+ // change.
+ Map props = alterTbl.getProps();
+ if (props.size() <= 1 &&
props.get(ReplicationSpec.KEY.CURR_STATE_ID.toString()) != null) {
+isTxn = false;
+ } else {
+isTxn = true;
+ }
+}
+// TODO: Somehow we have to signal alterPartitions that it's part of
replication and
+// should use replication's valid writeid list instead of creating
one.
Review comment:
I have addressed this comment and removed it as well. But didn't commit the
change and thus wasn't part of the PR. I have updated PR. This TODO is no more
there. Sorry.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 219176)
Time Spent: 2h 40m (was: 2.5h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802496#comment-16802496
]
Hive QA commented on HIVE-21230:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 3 new + 10 unchanged - 0 fixed
= 13 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16705/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16705/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16705/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java
[jira] [Commented] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802494#comment-16802494 ] Miklos Gergely commented on HIVE-21517: --- [~thejas] please take a look at this small patch, merge if you agree. > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Status: Open (was: Patch Available)
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Attachment: HIVE-21231.05.patch
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Status: Patch Available (was: Open)
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch, HIVE-21231.05.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802482#comment-16802482
]
Hive QA commented on HIVE-21518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963808/HIVE-21518.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15846 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_notequalns]
(batchId=153)
org.apache.hadoop.hive.llap.metrics.TestReadWriteLockMetrics.testWithoutContention
(batchId=330)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16704/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16704/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16704/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963808 - PreCommit-HIVE-Build
> GenericUDFOPNotEqualNS does not run in LLAP
> ---
>
> Key: HIVE-21518
> URL: https://issues.apache.org/jira/browse/HIVE-21518
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-21518.1.patch
>
>
> GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP
> mode, because it is not registered as a built-in function.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802467#comment-16802467 ] Ashutosh Chauhan commented on HIVE-21517: - +1 > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[
https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802463#comment-16802463
]
Hive QA commented on HIVE-21518:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16704/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16704/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> GenericUDFOPNotEqualNS does not run in LLAP
> ---
>
> Key: HIVE-21518
> URL: https://issues.apache.org/jira/browse/HIVE-21518
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-21518.1.patch
>
>
> GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP
> mode, because it is not registered as a built-in function.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21517) Fix AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802448#comment-16802448
]
Hive QA commented on HIVE-21517:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963797/HIVE-21517.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16703/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16703/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16703/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963797 - PreCommit-HIVE-Build
> Fix AggregateStatsCache
> ---
>
> Key: HIVE-21517
> URL: https://issues.apache.org/jira/browse/HIVE-21517
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21517.01.patch
>
>
> Due to a bug AggregateStatsCache is not returning the best matching result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21517) Fix AggregateStatsCache
[
https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802431#comment-16802431
]
Hive QA commented on HIVE-21517:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
21s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16703/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16703/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix AggregateStatsCache
> ---
>
> Key: HIVE-21517
> URL: https://issues.apache.org/jira/browse/HIVE-21517
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21517.01.patch
>
>
> Due to a bug AggregateStatsCache is not returning the best matching result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[
https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802397#comment-16802397
]
Hive QA commented on HIVE-21516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963796/HIVE-21516.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16702/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16702/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16702/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963796 - PreCommit-HIVE-Build
> Fix spark downloading for q tests
> -
>
> Key: HIVE-21516
> URL: https://issues.apache.org/jira/browse/HIVE-21516
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21516.01.patch
>
>
> Currently itests/pom.xml declares a command to generated the download script
> for spark, thus it is re-generated every time any maven command is executed
> for any sub project of itests. AS a side effect it is leaving download.sh
> files everywhere. The download.sh file is almost totally static, no need to
> recreate it every time, just requires $spark.version as a parameter.
> Also it is only working properly under linux, as it relies on the md5sum
> program which is not present in OS X. This means that if the spark tarball is
> partially downloaded on OS X, then it would never be re-downloaded. This
> should be fixed by making it work as well using md5 on OS X.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
[
https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802383#comment-16802383
]
Nitin commented on HIVE-14719:
--
[~lirui] Do we have any update on this issue ?
> ASTNode rootNode is not maintained properly when changing child/parent
> relation
> ---
>
> Key: HIVE-14719
> URL: https://issues.apache.org/jira/browse/HIVE-14719
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14719.1.patch, HIVE-14719.2.patch
>
>
> When I run some query like:
> {code}
> set hive.cbo.enable=false;
> select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
> {code}
> It gets error like:
> {noformat}
> FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
> 'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
> (tok_table_or_col sq_1) sq_corr_1))
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21516) Fix spark downloading for q tests
[
https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802375#comment-16802375
]
Hive QA commented on HIVE-21516:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16702/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16702/yetus/whitespace-eol.txt
|
| modules | C: itests itests/hive-unit itests/qtest-spark U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16702/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix spark downloading for q tests
> -
>
> Key: HIVE-21516
> URL: https://issues.apache.org/jira/browse/HIVE-21516
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21516.01.patch
>
>
> Currently itests/pom.xml declares a command to generated the download script
> for spark, thus it is re-generated every time any maven command is executed
> for any sub project of itests. AS a side effect it is leaving download.sh
> files everywhere. The download.sh file is almost totally static, no need to
> recreate it every time, just requires $spark.version as a parameter.
> Also it is only working properly under linux, as it relies on the md5sum
> program which is not present in OS X. This means that if the spark tarball is
> partially downloaded on OS X, then it would never be re-downloaded. This
> should be fixed by making it work as well using md5 on OS X.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21515) Improvement to MoveTrash Facilities
[
https://issues.apache.org/jira/browse/HIVE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802354#comment-16802354
]
Hive QA commented on HIVE-21515:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963787/HIVE-21515.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 324 failed/errored test(s), 15845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_move]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_only]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_part]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_nonstd_partitions_loc]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_format_part]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[rcfile_nonstd_partitions_loc]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[zero_rows_blobstore]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[zero_rows_hdfs]
(batchId=278)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_char1] (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2_orc]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_3]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table_h23]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade]
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde2]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_varchar1]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_gby2_map_multi_distinct]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udaf_percentile_approx_23]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_1] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[concatenate_inherit_table_location]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_merge_compressed]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_1] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_query3]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_2] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby10] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_multi_distinct]
(batchId=41)
org.apache.ha
[jira] [Commented] (HIVE-21515) Improvement to MoveTrash Facilities
[
https://issues.apache.org/jira/browse/HIVE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802336#comment-16802336
]
Hive QA commented on HIVE-21515:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
34s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 204 unchanged - 1
fixed = 204 total (was 205) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16701/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16701/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16701/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improvement to MoveTrash Facilities
> ---
>
> Key: HIVE-21515
> URL: https://issues.apache.org/jira/browse/HIVE-21515
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21515.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if
[
https://issues.apache.org/jira/browse/HIVE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802309#comment-16802309
]
Hive QA commented on HIVE-21152:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963778/HIVE-21152.05.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15845 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explode_null]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_write_correct_definition_levels]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf6] (batchId=57)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=182)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=149)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16699/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16699/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16699/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963778 - PreCommit-HIVE-Build
> Rewrite if expression to case and recognize simple case as an if
>
>
> Key: HIVE-21152
> URL: https://issues.apache.org/jira/browse/HIVE-21152
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21152.01.patch, HIVE-21152.02.patch,
> HIVE-21152.03.patch, HIVE-21152.04.patch, HIVE-21152.05.patch
>
>
> * {{IF}} is not part of the sql standard; however given its special form its
> simpler - and currently in Hive it also has vectorized support
> * people writing standard sql may write: {{CASE WHEN member=1 THEN attr+1
> else attr+2 END}} which is essentially an if.
> The idea is to rewrite IFs to CASEs for the cbo; and recognize simple
> "CASE"-s as IFs to get vectorization on them if possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802313#comment-16802313
]
Hive QA commented on HIVE-21231:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963779/HIVE-21231.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16700/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16700/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16700/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12963779/HIVE-21231.04.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963779 - PreCommit-HIVE-Build
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802307#comment-16802307
]
Vineet Garg commented on HIVE-21231:
[~mgergely] Whenever this patch is ready create a RB/pull request. I can take a
look at it.
Also FYI HIVE-21230 touches the same part of the code.
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802302#comment-16802302 ] Vineet Garg commented on HIVE-21518: +1 pending tests > GenericUDFOPNotEqualNS does not run in LLAP > --- > > Key: HIVE-21518 > URL: https://issues.apache.org/jira/browse/HIVE-21518 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21518.1.patch > > > GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP > mode, because it is not registered as a built-in function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Attachment: HIVE-21230.2.patch
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Status: Patch Available (was: Open)
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21230) LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side (HiveJoinAddNotNullRule bails out for outer joins)
[
https://issues.apache.org/jira/browse/HIVE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-21230:
---
Status: Open (was: Patch Available)
> LEFT OUTER JOIN does not generate transitive IS NOT NULL filter on right side
> (HiveJoinAddNotNullRule bails out for outer joins)
>
>
> Key: HIVE-21230
> URL: https://issues.apache.org/jira/browse/HIVE-21230
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21230.1.patch, HIVE-21230.2.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> LEFT JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 = t1.col0 AND t0.col1 = t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null in the right input and
> introduce the corresponding filter predicate. Currently, the rule just bails
> out if it is not an inner join.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java#L79
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19638) Configuration not passed to ORC Reader.Options
[ https://issues.apache.org/jira/browse/HIVE-19638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-19638: - Resolution: Duplicate Status: Resolved (was: Patch Available) This was fixed in master. I also pushed it back to 2.3 and 3.1. > Configuration not passed to ORC Reader.Options > -- > > Key: HIVE-19638 > URL: https://issues.apache.org/jira/browse/HIVE-19638 > Project: Hive > Issue Type: Bug > Components: Configuration, File Formats, ORC >Affects Versions: 2.3.0, 2.3.1, 2.3.2 >Reporter: Rentao Wu >Assignee: Rentao Wu >Priority: Major > Attachments: HIVE-19638.patch > > > Configuration is not passed to ORC's Reader.Option in OrcFileInputFormat > which causes some [ORC > configurations|https://github.com/apache/orc/blob/master/java/core/src/java/org/apache/orc/Reader.java#L170-L176] > to not be able to be picked up. > Related issues: > For example, the ORC upgrade in Hive 2.3.x changed schema evolution from > positional to column name matching. A backwards compatibility configuration > "orc.force.positional.evolution" could be set in ORC Reader.Options by > [ORC-120|https://issues.apache.org/jira/browse/ORC-120] however it could not > be picked up resulting in null values when querying ORC tables where the > column names do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-21518: -- Status: Patch Available (was: Open) [~vgarg] can you review? This just adds GenericUDFOPNotEqualNS as a hidden built-in function). > GenericUDFOPNotEqualNS does not run in LLAP > --- > > Key: HIVE-21518 > URL: https://issues.apache.org/jira/browse/HIVE-21518 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21518.1.patch > > > GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP > mode, because it is not registered as a built-in function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-21518: - > GenericUDFOPNotEqualNS does not run in LLAP > --- > > Key: HIVE-21518 > URL: https://issues.apache.org/jira/browse/HIVE-21518 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21518.1.patch > > > GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP > mode, because it is not registered as a built-in function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if
[
https://issues.apache.org/jira/browse/HIVE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802292#comment-16802292
]
Hive QA commented on HIVE-21152:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16699/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16699/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Rewrite if expression to case and recognize simple case as an if
>
>
> Key: HIVE-21152
> URL: https://issues.apache.org/jira/browse/HIVE-21152
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21152.01.patch, HIVE-21152.02.patch,
> HIVE-21152.03.patch, HIVE-21152.04.patch, HIVE-21152.05.patch
>
>
> * {{IF}} is not part of the sql standard; however given its special form its
> simpler - and currently in Hive it also has vectorized support
> * people writing standard sql may write: {{CASE WHEN member=1 THEN attr+1
> else attr+2 END}} which is essentially an if.
> The idea is to rewrite IFs to CASEs for the cbo; and recognize simple
> "CASE"-s as IFs to get vectorization on them if possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21518) GenericUDFOPNotEqualNS does not run in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-21518: -- Attachment: HIVE-21518.1.patch > GenericUDFOPNotEqualNS does not run in LLAP > --- > > Key: HIVE-21518 > URL: https://issues.apache.org/jira/browse/HIVE-21518 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-21518.1.patch > > > GenericUDFOPNotEqualNS (Not equal nullsafe operator) does not run in LLAP > mode, because it is not registered as a built-in function. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20126) OrcInputFormat does not pass conf to orc reader options
[ https://issues.apache.org/jira/browse/HIVE-20126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802290#comment-16802290 ] Owen O'Malley commented on HIVE-20126: -- Also committed back to branch-3.1, branch-2, and branch-2.3. > OrcInputFormat does not pass conf to orc reader options > --- > > Key: HIVE-20126 > URL: https://issues.apache.org/jira/browse/HIVE-20126 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Prasanth Jayachandran >Priority: Critical > Fix For: 2.4.0, 4.0.0, 3.2.0, 2.3.4, 3.1.2 > > Attachments: HIVE-20126.1.patch > > > VectorizedOrcInputFormat creates Orc reader options without passing in the > configuration object. Without it setting orc configurations will not have any > impact. > Example: > set orc.force.positional.evolution=true; > does not work for positional schema evolution (will attach test case). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20126) OrcInputFormat does not pass conf to orc reader options
[ https://issues.apache.org/jira/browse/HIVE-20126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-20126: - Fix Version/s: 2.3.4 3.1.2 2.4.0 > OrcInputFormat does not pass conf to orc reader options > --- > > Key: HIVE-20126 > URL: https://issues.apache.org/jira/browse/HIVE-20126 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 3.2.0 >Reporter: Aswathy Chellammal Sreekumar >Assignee: Prasanth Jayachandran >Priority: Critical > Fix For: 2.4.0, 4.0.0, 3.2.0, 2.3.4, 3.1.2 > > Attachments: HIVE-20126.1.patch > > > VectorizedOrcInputFormat creates Orc reader options without passing in the > configuration object. Without it setting orc configurations will not have any > impact. > Example: > set orc.force.positional.evolution=true; > does not work for positional schema evolution (will attach test case). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802273#comment-16802273
]
Hive QA commented on HIVE-21231:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963779/HIVE-21231.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16698/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16698/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16698/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-26 23:30:54.498
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16698/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-26 23:30:54.502
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
0b8cfa7..298f749 branch-2 -> origin/branch-2
+ git reset --hard HEAD
HEAD is now at 5708a0b HIVE-21479: NPE during metastore cache update (Daniel
Dai, reviewed by Zoltan Haindrich)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5708a0b HIVE-21479: NPE during metastore cache update (Daniel
Dai, reviewed by Zoltan Haindrich)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-26 23:30:57.435
+ rm -rf ../yetus_PreCommit-HIVE-Build-16698
+ mkdir ../yetus_PreCommit-HIVE-Build-16698
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16698
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16698/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinAddNotNullRule.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRulesRegistry.java:
does not exist in index
error: a/ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/interval_3.q.out: does not exist in
index
error: a/ql/src/test/results/clientpositive/join43.q.out: does not exist in
index
error: a/ql/src/test/results/clientpositive/join_merging.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/llap/cross_prod_1.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/llap/semijoin.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/llap/subquery_corr.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/subquery_in.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/subquery_notin.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/subquery_scalar.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/llap/subquery_select.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/perf/spark/query1.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/perf/spark/query23.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/perf/spark/query24.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/perf/spark/query30.q.out: does not
exist in i
[jira] [Commented] (HIVE-21512) Upgrade jms-api to 2.0.2
[
https://issues.apache.org/jira/browse/HIVE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802270#comment-16802270
]
Hive QA commented on HIVE-21512:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963768/HIVE-21512.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15839 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration.org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigration
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16697/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16697/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16697/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963768 - PreCommit-HIVE-Build
> Upgrade jms-api to 2.0.2
>
>
> Key: HIVE-21512
> URL: https://issues.apache.org/jira/browse/HIVE-21512
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21512.01.patch
>
>
> I've noticed that for some time that sometimes there are issues with
> javax.jms:jms:1.1 artifact - because it doesn't seem to be available from
> maven central for some reason;
> https://issues.sonatype.org/browse/MVNCENTRAL-4708
> Alternatively; I think we might try to just upgrade to 2.0.2 version of the
> jms-api.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21517: -- Attachment: HIVE-21517.01.patch > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21517: -- Status: Patch Available (was: Open) > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21517.01.patch > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21517) Fix AggregateStatsCache
[ https://issues.apache.org/jira/browse/HIVE-21517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21517: - > Fix AggregateStatsCache > --- > > Key: HIVE-21517 > URL: https://issues.apache.org/jira/browse/HIVE-21517 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Due to a bug AggregateStatsCache is not returning the best matching result. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21512) Upgrade jms-api to 2.0.2
[
https://issues.apache.org/jira/browse/HIVE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802262#comment-16802262
]
Hive QA commented on HIVE-21512:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16697/dev-support/hive-personality.sh
|
| git revision | master / 5708a0b |
| Default Java | 1.8.0_111 |
| modules | C: hcatalog/server-extensions . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16697/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade jms-api to 2.0.2
>
>
> Key: HIVE-21512
> URL: https://issues.apache.org/jira/browse/HIVE-21512
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21512.01.patch
>
>
> I've noticed that for some time that sometimes there are issues with
> javax.jms:jms:1.1 artifact - because it doesn't seem to be available from
> maven central for some reason;
> https://issues.sonatype.org/browse/MVNCENTRAL-4708
> Alternatively; I think we might try to just upgrade to 2.0.2 version of the
> jms-api.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Status: Patch Available (was: Open) > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21516: -- Attachment: HIVE-21516.01.patch > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21516.01.patch > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21516) Fix spark downloading for q tests
[ https://issues.apache.org/jira/browse/HIVE-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21516: - > Fix spark downloading for q tests > - > > Key: HIVE-21516 > URL: https://issues.apache.org/jira/browse/HIVE-21516 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Currently itests/pom.xml declares a command to generated the download script > for spark, thus it is re-generated every time any maven command is executed > for any sub project of itests. AS a side effect it is leaving download.sh > files everywhere. The download.sh file is almost totally static, no need to > recreate it every time, just requires $spark.version as a parameter. > Also it is only working properly under linux, as it relies on the md5sum > program which is not present in OS X. This means that if the spark tarball is > partially downloaded on OS X, then it would never be re-downloaded. This > should be fixed by making it work as well using md5 on OS X. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802230#comment-16802230
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963763/HIVE-21304.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15842 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_buckets] (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_historical_timestamp]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=191)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket4]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=147)
org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1
(batchId=261)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookie (batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16696/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16696/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16696/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963763 - PreCommit-HIVE-Build
> Show Bucketing version for ReduceSinkOp in explain extended plan
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>
> cc [~vgarg]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Deleted] (HIVE-21514) Map data
[ https://issues.apache.org/jira/browse/HIVE-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lambertus deleted HIVE-21514: --- > Map data > > > Key: HIVE-21514 > URL: https://issues.apache.org/jira/browse/HIVE-21514 > Project: Hive > Issue Type: Bug >Reporter: Simon poortman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lambertus updated HIVE-21504:
---
Comment: was deleted
(was: Key)
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lambertus reassigned HIVE-21504:
--
Assignee: (was: Simon poortman)
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lambertus updated HIVE-21504:
---
Comment: was deleted
(was: We are trying to use TypeUTILS TO VALIDATE THE DATA TYPE INFORMATION OF A
STRING)
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lambertus updated HIVE-21504:
---
Comment: was deleted
(was: Activiste the train to my map)
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802207#comment-16802207
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
34s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 2 new + 989 unchanged - 3
fixed = 991 total (was 992) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16696/dev-support/hive-personality.sh
|
| git revision | master / 10dfb15 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16696/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16696/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Show Bucketing version for ReduceSinkOp in explain extended plan
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch
>
>
> Show Bucketing version for ReduceSinkOp in explain extended plan.
> This helps identify what hashing algorithm is being used by by ReduceSinkOp.
>
> cc [~vgarg]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21479) NPE during metastore cache update
[
https://issues.apache.org/jira/browse/HIVE-21479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-21479:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Patch pushed to master. Thanks Zoltan for review!
> NPE during metastore cache update
> -
>
> Key: HIVE-21479
> URL: https://issues.apache.org/jira/browse/HIVE-21479
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21479.1.patch
>
>
> Saw the following stack during a long periodical update:
> {code}
> 2019-03-12T10:01:43,015 ERROR [CachedStore-CacheUpdateService: Thread-36]
> cache.CachedStore: Update failure:java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.updateTableColStats(CachedStore.java:508)
> at
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.update(CachedStore.java:461)
> at
> org.apache.hadoop.hive.metastore.cache.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:396)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The reason is we get the table list at very early stage and then refresh
> table one by one. It is likely table is removed during the interim. We need
> to deal with this case during cache update.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21478) Metastore cache update shall capture exception
[ https://issues.apache.org/jira/browse/HIVE-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21478: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Patch pushed to master. Thanks Zoltan for review! > Metastore cache update shall capture exception > -- > > Key: HIVE-21478 > URL: https://issues.apache.org/jira/browse/HIVE-21478 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21478.1.patch > > > We definitely need to capture any exception during > CacheUpdateMasterWork.update(), otherwise, Java would refuse to schedule > future update(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21507) Hive swallows NPE if no delegation token found
[ https://issues.apache.org/jira/browse/HIVE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21507: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) +1. Patch pushed to master. Thanks Denes! > Hive swallows NPE if no delegation token found > -- > > Key: HIVE-21507 > URL: https://issues.apache.org/jira/browse/HIVE-21507 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 3.1.1 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Labels: usability > Fix For: 4.0.0 > > Attachments: HIVE-21507.001.patch, HIVE-21507.002.patch, > HIVE-21507.003.patch > > > In case if there is no delegation token put into token file, this > [line|https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L777] > will cause a NullPointerException which is not handled and the user is not > notified in any way. > To cause NPE the use case is to have an Oozie Sqoop import to Hive in a > kerberized cluster. Oozie puts the delegation token into the token file with > id: *HIVE_DELEGATION_TOKEN_hiveserver2ClientToken*. So with id *hive* it is > not working. However, fallback code uses the key which Oozie provides > [this|https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L784] > way. > I suggest to have warning message to user that key with id *hive* cannot be > used and falling back to get delegation token from the session. > I am creating the patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21305) LLAP: Option to skip cache for ETL queries
[ https://issues.apache.org/jira/browse/HIVE-21305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-21305: - Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks Gopal for the review! > LLAP: Option to skip cache for ETL queries > -- > > Key: HIVE-21305 > URL: https://issues.apache.org/jira/browse/HIVE-21305 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21305.1.patch, HIVE-21305.2.patch, > HIVE-21305.3.patch > > > To avoid ETL queries from polluting the cache, would be good to detect such > queries at compile time and optional skip llap io for such queries. > org.apache.hadoop.hive.ql.parse.QBParseInfo.hasInsertTables() is the simplest > way to catch ETL queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21305) LLAP: Option to skip cache for ETL queries
[ https://issues.apache.org/jira/browse/HIVE-21305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802192#comment-16802192 ] Prasanth Jayachandran commented on HIVE-21305: -- [~gopalv] LLAP sets ROWS_EMITTED as VRB batch size, whereas tez counts VRB as 1 record, hence the difference. [~rajesh.balamohan] This disables read through cache behavior for ETL queries reading from text table and writing elsewhere (which could be temp table as well). > LLAP: Option to skip cache for ETL queries > -- > > Key: HIVE-21305 > URL: https://issues.apache.org/jira/browse/HIVE-21305 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21305.1.patch, HIVE-21305.2.patch, > HIVE-21305.3.patch > > > To avoid ETL queries from polluting the cache, would be good to detect such > queries at compile time and optional skip llap io for such queries. > org.apache.hadoop.hive.ql.parse.QBParseInfo.hasInsertTables() is the simplest > way to catch ETL queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17395) HiveServer2 parsing a command with a lot of "("
[
https://issues.apache.org/jira/browse/HIVE-17395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802180#comment-16802180
]
Julian Hyde commented on HIVE-17395:
[~kgyrtkirk] Thanks. I was aware of HIVE-15388 but it does look as if
HIVE-18624 is a better match. This might be a duplicate. It does match the
timescale when this issue appeared.
> HiveServer2 parsing a command with a lot of "("
> ---
>
> Key: HIVE-17395
> URL: https://issues.apache.org/jira/browse/HIVE-17395
> Project: Hive
> Issue Type: Bug
> Components: Beeline, HiveServer2
>Affects Versions: 2.3.0
>Reporter: dan young
>Priority: Major
>
> Hello,
> We're seeing what appears to be the same issue that was outlined in
> HIVE-15388 where the query parser spends a lot of time (never returns and I
> need to kill the beeline process) parsing a command with a lot of "(" . I
> tried this in both 2.2 and now 2.3.
> Here's an example query (this is auto generated SQL BTW) in beeline that
> never completes/parses, I end up just killing the beeline process.
> It looks like something similar was addressed as part of HIVE-15388. Any
> ideas on how to address this? write better SQL? patch?
> Regards,
> Dano
> {noformat}
> Connected to: Apache Hive (version 2.3.0)
> Driver: Hive JDBC (version 2.3.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 2.3.0 by Apache Hive
> 0: jdbc:hive2://localhost:1/test_db> SELECT
> ((UNIX_TIMESTAMP(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP(CONCAT(ADD_MONTHS(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 AS STRING), '-',
> LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING),
> 2, '0'), '-01 00:00:00') AS TIMESTAMP),
> 1),SUBSTRING(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 AS STRING), '-',
> LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING),
> 2, '0'), '-01 00:00:00') AS TIMESTAMP),11))), 'MM'))),
> -3),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP(CONCAT(ADD_MONTHS(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 AS STRING), '-',
> LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING),
> 2, '0'), '-01 00:00:00') AS TIMESTAMP),
> 1),SUBSTRING(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 AS STRING), '-',
> LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),
> -1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2012-04-20
> 00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING),
> 2, '0'), '-01 00:00:00') AS TIMESTAMP),11))), 'MM'))),11));
> When I did a jstack on the HiveServer2, it appears the be stuck/running in
> the HiveParser/antlr.
> "e62658bd-5ea9-43c4-898f-3048d913f192 HiveServer2-Handler-Pool: Thread-96"
> #96 prio=5 os_prio=0 tid=0x7fb78c366000 nid=0x4476 runnable
> [0x7fb77d7bb000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA36.specialStateTransition(HiveParser_IdentifiersParser.java:31502)
> at org.antlr.runtime.DFA.predict(DFA.java:80)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6746)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6988)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnary
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802177#comment-16802177
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963742/HIVE-21001.46.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16695/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16695/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16695/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12963742/HIVE-21001.46.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963742 - PreCommit-HIVE-Build
> Upgrade to calcite-1.19
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch, HIVE-21001.06.patch,
> HIVE-21001.07.patch, HIVE-21001.08.patch, HIVE-21001.08.patch,
> HIVE-21001.08.patch, HIVE-21001.09.patch, HIVE-21001.09.patch,
> HIVE-21001.09.patch, HIVE-21001.10.patch, HIVE-21001.11.patch,
> HIVE-21001.12.patch, HIVE-21001.13.patch, HIVE-21001.15.patch,
> HIVE-21001.16.patch, HIVE-21001.17.patch, HIVE-21001.18.patch,
> HIVE-21001.18.patch, HIVE-21001.19.patch, HIVE-21001.20.patch,
> HIVE-21001.21.patch, HIVE-21001.22.patch, HIVE-21001.22.patch,
> HIVE-21001.22.patch, HIVE-21001.23.patch, HIVE-21001.24.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.26.patch,
> HIVE-21001.26.patch, HIVE-21001.26.patch, HIVE-21001.27.patch,
> HIVE-21001.28.patch, HIVE-21001.29.patch, HIVE-21001.29.patch,
> HIVE-21001.30.patch, HIVE-21001.31.patch, HIVE-21001.32.patch,
> HIVE-21001.34.patch, HIVE-21001.35.patch, HIVE-21001.36.patch,
> HIVE-21001.37.patch, HIVE-21001.38.patch, HIVE-21001.39.patch,
> HIVE-21001.40.patch, HIVE-21001.41.patch, HIVE-21001.42.patch,
> HIVE-21001.43.patch, HIVE-21001.44.patch, HIVE-21001.45.patch,
> HIVE-21001.45.patch, HIVE-21001.46.patch
>
>
> XLEAR LIBRARY CACHE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[
https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802175#comment-16802175
]
Hive QA commented on HIVE-21316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963741/HIVE-21316.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 221 failed/errored test(s), 15841 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_cascade]
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_multi]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_ppd_non_deterministic]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_outer_join_ppr]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_union_view]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine2] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine3] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropagateForSubQuery]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantfolding]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_or_replace_view]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view] (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view_partitioned]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_1] (batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_2] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_4] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_6] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_1] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_2] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_formatted_view_partitioned]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_formatted_view_partitioned_json]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_dependency]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr_multi_distinct]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_dyn_part]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input25] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input26] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_view] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[keep_uniform]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_8]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part14]
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_4] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge4] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[metadataOnlyOptimizer]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_boolean]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_char]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_is_null]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_join2]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_boolean]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_char]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_date]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.te
[jira] [Commented] (HIVE-21515) Improvement to MoveTrash Facilities
[ https://issues.apache.org/jira/browse/HIVE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802167#comment-16802167 ] David Mollitor commented on HIVE-21515: --- Suggested improvements are based on the results of a series of stack traces from a production system when a set of files could not be deleted because of permissions issues. > Improvement to MoveTrash Facilities > --- > > Key: HIVE-21515 > URL: https://issues.apache.org/jira/browse/HIVE-21515 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-21515.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21515) Improvement to MoveTrash Facilities
[ https://issues.apache.org/jira/browse/HIVE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-21515: -- Attachment: HIVE-21515.1.patch > Improvement to MoveTrash Facilities > --- > > Key: HIVE-21515 > URL: https://issues.apache.org/jira/browse/HIVE-21515 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-21515.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21515) Improvement to MoveTrash Facilities
[ https://issues.apache.org/jira/browse/HIVE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-21515: -- Status: Patch Available (was: Open) > Improvement to MoveTrash Facilities > --- > > Key: HIVE-21515 > URL: https://issues.apache.org/jira/browse/HIVE-21515 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-21515.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21515) Improvement to MoveTrash Facilities
[ https://issues.apache.org/jira/browse/HIVE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-21515: - > Improvement to MoveTrash Facilities > --- > > Key: HIVE-21515 > URL: https://issues.apache.org/jira/browse/HIVE-21515 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0, 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-21515.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21504 started by Simon poortman.
-
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21513) ACID: Running merge concurrently with minor compaction causes a later select * to throw exception
[ https://issues.apache.org/jira/browse/HIVE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21513: Affects Version/s: 4.0.0 > ACID: Running merge concurrently with minor compaction causes a later select > * to throw exception > -- > > Key: HIVE-21513 > URL: https://issues.apache.org/jira/browse/HIVE-21513 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0, 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Repro steps: > - Create table > - Load some data > - Run merge so records gets updated and delete_delta dirs are created > - Manually initiate minor compaction: ALTER TABLE ... COMPACT 'minor'; > - While the compaction is running keep executing the merge statement > - After some time try to do simple select *; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21513) ACID: Running merge concurrently with minor compaction causes a later select * to throw exception
[ https://issues.apache.org/jira/browse/HIVE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-21513: --- Assignee: Vaibhav Gumashta > ACID: Running merge concurrently with minor compaction causes a later select > * to throw exception > -- > > Key: HIVE-21513 > URL: https://issues.apache.org/jira/browse/HIVE-21513 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Repro steps: > - Create table > - Load some data > - Run merge so records gets updated and delete_delta dirs are created > - Manually initiate minor compaction: ALTER TABLE ... COMPACT 'minor'; > - While the compaction is running keep executing the merge statement > - After some time try to do simple select *; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21316) Comparision of varchar column and string literal should happen in varchar
[
https://issues.apache.org/jira/browse/HIVE-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802130#comment-16802130
]
Hive QA commented on HIVE-21316:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
36s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 6 new + 137 unchanged - 0
fixed = 143 total (was 137) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 9 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
47s{color} | {color:red} ql generated 1 new + 2255 unchanged - 0 fixed = 2256
total (was 2255) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.translator.RexNodeConverter$HiveNlsString
doesn't override org.apache.calcite.util.NlsString.equals(Object) At
RexNodeConverter.java:At RexNodeConverter.java:[line 1] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16694/dev-support/hive-personality.sh
|
| git revision | master / 866498a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16694/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16694/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16694/yetus/new-findbugs-ql.html
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16694/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Comparision of varchar column and string literal should happen in varchar
> -
>
> Key: HIVE-21316
> URL: https://issues.apache.org/jira/browse/HIVE-21316
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21316.01.patch, HIVE-21316.02.patch,
> HIVE-21316.03.patch, HIVE-21316.04.patch
>
>
> this is most probably the root cause behind HIVE-21310 as well
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21503) Vectorization: query with regex gives incorrect results with vectorization
[
https://issues.apache.org/jira/browse/HIVE-21503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802123#comment-16802123
]
Gopal V commented on HIVE-21503:
That definitely looks like the CASE vectorization bug.
Confirm by testing with - HIVE-18529
> Vectorization: query with regex gives incorrect results with vectorization
> --
>
> Key: HIVE-21503
> URL: https://issues.apache.org/jira/browse/HIVE-21503
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Rajesh Balamohan
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21503.01.WIP.patch
>
>
> i see wrong results with vectorization. Without vectorization, it works fine.
> {noformat}
> e.g
> WHEN x like '%radio%' THEN 'radio'
> WHEN x like '%tv%' THEN 'tv'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
[ https://issues.apache.org/jira/browse/HIVE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21290: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks [~klcopp]. Patch does not apply cleanly to branch-3 and branch-3.1. Can you upload the patches for those branches so we apply the fix there too? > Restore historical way of handling timestamps in Parquet while keeping the > new semantics at the same time > - > > Key: HIVE-21290 > URL: https://issues.apache.org/jira/browse/HIVE-21290 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Ivanfi >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21290.1.patch, HIVE-21290.2.patch, > HIVE-21290.2.patch, HIVE-21290.3.patch, HIVE-21290.4.patch, > HIVE-21290.4.patch, HIVE-21290.5.patch > > > This sub-task is for implementing the Parquet-specific parts of the following > plan: > h1. Problem > Historically, the semantics of the TIMESTAMP type in Hive depended on the > file format. Timestamps in Avro, Parquet and RCFiles with a binary SerDe had > _Instant_ semantics, while timestamps in ORC, textfiles and RCFiles with a > text SerDe had _LocalDateTime_ semantics. > The Hive community wanted to get rid of this inconsistency and have > _LocalDateTime_ semantics in Avro, Parquet and RCFiles with a binary SerDe as > well. *Hive 3.1 turned off normalization to UTC* to achieve this. While this > leads to the desired new semantics, it also leads to incorrect results when > new Hive versions read timestamps written by old Hive versions or when old > Hive versions or any other component not aware of this change (including > legacy Impala and Spark versions) read timestamps written by new Hive > versions. > h1. Solution > To work around this issue, Hive *should restore the practice of normalizing > to UTC* when writing timestamps to Avro, Parquet and RCFiles with a binary > SerDe. In itself, this would restore the historical _Instant_ semantics, > which is undesirable. In order to achieve the desired _LocalDateTime_ > semantics in spite of normalizing to UTC, newer Hive versions should record > the session-local local time zone in the file metadata fields serving > arbitrary key-value storage purposes. > When reading back files with this time zone metadata, newer Hive versions (or > any other new component aware of this extra metadata) can achieve > _LocalDateTime_ semantics by *converting from UTC to the saved time zone > (instead of to the local time zone)*. Legacy components that are unaware of > the new metadata can read the files without any problem and the timestamps > will show the historical Instant behaviour to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21290) Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
[ https://issues.apache.org/jira/browse/HIVE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802101#comment-16802101 ] Jesus Camacho Rodriguez commented on HIVE-21290: +1 > Restore historical way of handling timestamps in Parquet while keeping the > new semantics at the same time > - > > Key: HIVE-21290 > URL: https://issues.apache.org/jira/browse/HIVE-21290 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Ivanfi >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21290.1.patch, HIVE-21290.2.patch, > HIVE-21290.2.patch, HIVE-21290.3.patch, HIVE-21290.4.patch, > HIVE-21290.4.patch, HIVE-21290.5.patch > > > This sub-task is for implementing the Parquet-specific parts of the following > plan: > h1. Problem > Historically, the semantics of the TIMESTAMP type in Hive depended on the > file format. Timestamps in Avro, Parquet and RCFiles with a binary SerDe had > _Instant_ semantics, while timestamps in ORC, textfiles and RCFiles with a > text SerDe had _LocalDateTime_ semantics. > The Hive community wanted to get rid of this inconsistency and have > _LocalDateTime_ semantics in Avro, Parquet and RCFiles with a binary SerDe as > well. *Hive 3.1 turned off normalization to UTC* to achieve this. While this > leads to the desired new semantics, it also leads to incorrect results when > new Hive versions read timestamps written by old Hive versions or when old > Hive versions or any other component not aware of this change (including > legacy Impala and Spark versions) read timestamps written by new Hive > versions. > h1. Solution > To work around this issue, Hive *should restore the practice of normalizing > to UTC* when writing timestamps to Avro, Parquet and RCFiles with a binary > SerDe. In itself, this would restore the historical _Instant_ semantics, > which is undesirable. In order to achieve the desired _LocalDateTime_ > semantics in spite of normalizing to UTC, newer Hive versions should record > the session-local local time zone in the file metadata fields serving > arbitrary key-value storage purposes. > When reading back files with this time zone metadata, newer Hive versions (or > any other new component aware of this extra metadata) can achieve > _LocalDateTime_ semantics by *converting from UTC to the saved time zone > (instead of to the local time zone)*. Legacy components that are unaware of > the new metadata can read the files without any problem and the timestamps > will show the historical Instant behaviour to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21507) Hive swallows NPE if no delegation token found
[
https://issues.apache.org/jira/browse/HIVE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802096#comment-16802096
]
Hive QA commented on HIVE-21507:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963744/HIVE-21507.003.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15842 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16693/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16693/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16693/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963744 - PreCommit-HIVE-Build
> Hive swallows NPE if no delegation token found
> --
>
> Key: HIVE-21507
> URL: https://issues.apache.org/jira/browse/HIVE-21507
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: HIVE-21507.001.patch, HIVE-21507.002.patch,
> HIVE-21507.003.patch
>
>
> In case if there is no delegation token put into token file, this
> [line|https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L777]
> will cause a NullPointerException which is not handled and the user is not
> notified in any way.
> To cause NPE the use case is to have an Oozie Sqoop import to Hive in a
> kerberized cluster. Oozie puts the delegation token into the token file with
> id: *HIVE_DELEGATION_TOKEN_hiveserver2ClientToken*. So with id *hive* it is
> not working. However, fallback code uses the key which Oozie provides
> [this|https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L784]
> way.
> I suggest to have warning message to user that key with id *hive* cannot be
> used and falling back to get delegation token from the session.
> I am creating the patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21497) Direct SQL exception thrown by PartitionManagementTask
[
https://issues.apache.org/jira/browse/HIVE-21497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21497:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for reviewing [~prasanth_j]!
> Direct SQL exception thrown by PartitionManagementTask
> --
>
> Key: HIVE-21497
> URL: https://issues.apache.org/jira/browse/HIVE-21497
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21497.3.patch
>
>
> Metastore runs background thread out of which one is partition discovery.
> While removing expired partitions following exception is thrown
> {code:java}
> 2019-03-24 04:24:59.583 WARN [PartitionDiscoveryTask-0]
> metastore.MetaStoreDirectSql: Failed to execute [select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner join
> "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 inner join
> "PARTITION_KEY_VALS" "FILTER1" on "FILTER1"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER1"."INTEGER_IDX" = 1 inner join
> "PARTITION_KEY_VALS" "FILTER2" on "FILTER2"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER2"."INTEGER_IDX" = 2 where
> "DBS"."CTLG_NAME" = ? and ( ( (((case when "FILTER0"."PART_KEY_VAL" <> ? and
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0
> then cast("FILTER0"."PART_KEY_VAL" as date) else null end) = ?) and
> ("FILTER1"."PART_KEY_VAL" = ?)) and ("FILTER2"."PART_KEY_VAL" = ?)) )] with
> parameters [logs, sys, hive, __HIVE_DEFAULT_PARTITION__, logs, sys, hive,
> 2019-03-23, warehouse-1553300821-692w, metastore-db-create-job]
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner join
> "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 inner join
> "PARTITION_KEY_VALS" "FILTER1" on "FILTER1"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER1"."INTEGER_IDX" = 1 inner join
> "PARTITION_KEY_VALS" "FILTER2" on "FILTER2"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER2"."INTEGER_IDX" = 2 where
> "DBS"."CTLG_NAME" = ? and ( ( (((case when "FILTER0"."PART_KEY_VAL" <> ? and
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0
> then cast("FILTER0"."PART_KEY_VAL" as date) else null end) = ?) and
> ("FILTER1"."PART_KEY_VAL" = ?)) and ("FILTER2"."PART_KEY_VAL" = ?)) )".
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2042)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:621)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:487)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:3426)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:3418)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3702)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3453)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:3406)
> at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy33.getPartitionsByExpr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partitions_req(HiveMetaStore.java:4521)
> at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
> at
> sun.re
[jira] [Updated] (HIVE-21496) Automatic sizing of unordered buffer can overflow
[
https://issues.apache.org/jira/browse/HIVE-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21496:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for reviewing [~prasanth_j]!
> Automatic sizing of unordered buffer can overflow
> -
>
> Key: HIVE-21496
> URL: https://issues.apache.org/jira/browse/HIVE-21496
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21496.01.patch, HIVE-21496.02.patch, hive.log
>
>
> HIVE-21329 added automatic sizing of tez unordered partitioned KV buffer
> based on group by statistics. However, some corner cases for group by
> statistics sets Long.MAX for data size. This ends up setting Integer.MAX for
> unordered KV buffer size. This buffer size is expected to be in MB.
> Converting Integer.MAX value from MB to bytes will overflow and following
> exception is thrown.
> {code:java}
> 2019-03-23T01:35:17,760 INFO [Dispatcher thread {Central}]
> HistoryEventHandler.criticalEvents:
> [HISTORY][DAG:dag_1553330105749_0001_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1553330105749_0001_1_00_00_0,
> creationTime=1553330117468, allocationTime=1553330117524,
> startTime=1553330117562, finishTime=1553330117755, timeTaken=193,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR,
> diagnostics=Error: Error while running task ( failure ) :
> attempt_1553330105749_0001_1_00_00_0:java.lang.IllegalArgumentException
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:110)
> at
> org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:214)
> at
> org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput.initialize(UnorderedPartitionedKVOutput.java:76)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:537)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:520)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:505)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>
> Stats for GBY operator is getting Long.MAX_VALUE as seen below
> {code:java}
> 2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-TS[0] (logs): numRows: 1795
> dataSize: 4443078 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 359 numNulls: 89
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true}
> 2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: Estimating row count for
> GenericUDFOPEqual(Column[severity], Const string ERROR) Original num rows:
> 1795 New num rows: 5
> 2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [1] STATS-FIL[8]: numRows: 5 dataSize:
> 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity=
> colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen:
> 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
> 2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.FilterOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats:
> PARTIAL Column stats: NONE) on: FIL[8]
> 2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.SelectOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats:
> PARTIAL Column stats: NONE) on: SEL[2]
> 2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcF
[jira] [Commented] (HIVE-21497) Direct SQL exception thrown by PartitionManagementTask
[
https://issues.apache.org/jira/browse/HIVE-21497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802082#comment-16802082
]
Prasanth Jayachandran commented on HIVE-21497:
--
lgtm, +1
> Direct SQL exception thrown by PartitionManagementTask
> --
>
> Key: HIVE-21497
> URL: https://issues.apache.org/jira/browse/HIVE-21497
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21497.3.patch
>
>
> Metastore runs background thread out of which one is partition discovery.
> While removing expired partitions following exception is thrown
> {code:java}
> 2019-03-24 04:24:59.583 WARN [PartitionDiscoveryTask-0]
> metastore.MetaStoreDirectSql: Failed to execute [select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner join
> "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 inner join
> "PARTITION_KEY_VALS" "FILTER1" on "FILTER1"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER1"."INTEGER_IDX" = 1 inner join
> "PARTITION_KEY_VALS" "FILTER2" on "FILTER2"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER2"."INTEGER_IDX" = 2 where
> "DBS"."CTLG_NAME" = ? and ( ( (((case when "FILTER0"."PART_KEY_VAL" <> ? and
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0
> then cast("FILTER0"."PART_KEY_VAL" as date) else null end) = ?) and
> ("FILTER1"."PART_KEY_VAL" = ?)) and ("FILTER2"."PART_KEY_VAL" = ?)) )] with
> parameters [logs, sys, hive, __HIVE_DEFAULT_PARTITION__, logs, sys, hive,
> 2019-03-23, warehouse-1553300821-692w, metastore-db-create-job]
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner join
> "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 inner join
> "PARTITION_KEY_VALS" "FILTER1" on "FILTER1"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER1"."INTEGER_IDX" = 1 inner join
> "PARTITION_KEY_VALS" "FILTER2" on "FILTER2"."PART_ID" =
> "PARTITIONS"."PART_ID" and "FILTER2"."INTEGER_IDX" = 2 where
> "DBS"."CTLG_NAME" = ? and ( ( (((case when "FILTER0"."PART_KEY_VAL" <> ? and
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0
> then cast("FILTER0"."PART_KEY_VAL" as date) else null end) = ?) and
> ("FILTER1"."PART_KEY_VAL" = ?)) and ("FILTER2"."PART_KEY_VAL" = ?)) )".
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2042)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:621)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:487)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:3426)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$9.getSqlResult(ObjectStore.java:3418)
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3702)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3453)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:3406)
> at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> at com.sun.proxy.$Proxy33.getPartitionsByExpr(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partitions_req(HiveMetaStore.java:4521)
> at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.j
[jira] [Commented] (HIVE-21496) Automatic sizing of unordered buffer can overflow
[
https://issues.apache.org/jira/browse/HIVE-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802081#comment-16802081
]
Prasanth Jayachandran commented on HIVE-21496:
--
lgtm, +1
> Automatic sizing of unordered buffer can overflow
> -
>
> Key: HIVE-21496
> URL: https://issues.apache.org/jira/browse/HIVE-21496
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21496.01.patch, HIVE-21496.02.patch, hive.log
>
>
> HIVE-21329 added automatic sizing of tez unordered partitioned KV buffer
> based on group by statistics. However, some corner cases for group by
> statistics sets Long.MAX for data size. This ends up setting Integer.MAX for
> unordered KV buffer size. This buffer size is expected to be in MB.
> Converting Integer.MAX value from MB to bytes will overflow and following
> exception is thrown.
> {code:java}
> 2019-03-23T01:35:17,760 INFO [Dispatcher thread {Central}]
> HistoryEventHandler.criticalEvents:
> [HISTORY][DAG:dag_1553330105749_0001_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1553330105749_0001_1_00_00_0,
> creationTime=1553330117468, allocationTime=1553330117524,
> startTime=1553330117562, finishTime=1553330117755, timeTaken=193,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR,
> diagnostics=Error: Error while running task ( failure ) :
> attempt_1553330105749_0001_1_00_00_0:java.lang.IllegalArgumentException
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:110)
> at
> org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:214)
> at
> org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput.initialize(UnorderedPartitionedKVOutput.java:76)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:537)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:520)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:505)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>
> Stats for GBY operator is getting Long.MAX_VALUE as seen below
> {code:java}
> 2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-TS[0] (logs): numRows: 1795
> dataSize: 4443078 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 359 numNulls: 89
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true}
> 2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: Estimating row count for
> GenericUDFOPEqual(Column[severity], Const string ERROR) Original num rows:
> 1795 New num rows: 5
> 2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [1] STATS-FIL[8]: numRows: 5 dataSize:
> 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity=
> colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen:
> 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
> 2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.FilterOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats:
> PARTIAL Column stats: NONE) on: FIL[8]
> 2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.SelectOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats:
> PARTIAL Column stats: NONE) on: SEL[2]
> 2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [1] STATS-SEL[2]: numRows: 5 dataSize:
> 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity=
>
[jira] [Updated] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21109: -- Labels: pull-request-available (was: ) > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218858
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269136269
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
##
@@ -1247,17 +1244,37 @@ private static void createReplImportTasks(
} else if (!replicationSpec.isMetadataOnly()
&& !shouldSkipDataCopyInReplScope(tblDesc, replicationSpec)) {
x.getLOG().debug("adding dependent CopyWork/MoveWork for table");
-t.addDependentTask(loadTable(fromURI, table,
replicationSpec.isReplace(),
-new Path(tblDesc.getLocation()), replicationSpec, x, writeId,
stmtId));
+dependentTasks = new ArrayList<>(1);
+dependentTasks.add(loadTable(fromURI, table,
replicationSpec.isReplace(),
+ new Path(tblDesc.getLocation()),
replicationSpec,
+ x, writeId, stmtId));
}
- if (dropTblTask != null) {
-// Drop first and then create
-dropTblTask.addDependentTask(t);
-x.getTasks().add(dropTblTask);
+ // During replication, by the time we reply a commit transaction event,
the table should
+ // have been already created when replaying previous events. So no need
to create table
+ // again. For some reason we need create table task for partitioned
table though.
Review comment:
The comment says for partitioned table, create table task needed but in the
code it is skipped always for commit txn event. Which one is correct?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218858)
Time Spent: 1h 10m (was: 1h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218855&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218855
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269156935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -1894,6 +1898,16 @@ private void create_table_core(final RawStore ms, final
Table tbl,
List checkConstraints)
throws AlreadyExistsException, MetaException,
InvalidObjectException, NoSuchObjectException, InvalidInputException {
+
+ ColumnStatistics colStats = null;
+ // If the given table has column statistics, save it here. We will
update it later.
+ // We don't want it to be part of the Table object being created, lest
the create table
Review comment:
Shall simplify the comment. "Column stats are not expected to be part of
Create table event and also shouldn't be persisted. So remove it from Table
object."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218855)
Time Spent: 50m (was: 40m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218867&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218867
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269247183
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestStatsReplicationScenarios.java
##
@@ -359,17 +383,20 @@ private void testStatsReplicationCommon(boolean
parallelBootstrap, boolean metad
}
@Test
- public void testForNonAcidTables() throws Throwable {
+ public void testNonParallelBootstrapLoad() throws Throwable {
+LOG.info("Testing " + testName.getClass().getName() + "." +
testName.getMethodName());
testStatsReplicationCommon(false, false);
}
@Test
- public void testForNonAcidTablesParallelBootstrapLoad() throws Throwable {
-testStatsReplicationCommon(true, false);
+ public void testForParallelBootstrapLoad() throws Throwable {
+LOG.info("Testing " + testName.getClass().getName() + "." +
testName.getMethodName());
+testStatsReplicationCommon(true, false );
}
@Test
- public void testNonAcidMetadataOnlyDump() throws Throwable {
+ public void testMetadataOnlyDump() throws Throwable {
Review comment:
Add more tests for the following scenarios.
1. REPL LOAD fails after replicating table or partition objects with stats
but before setting last replId. Now, retry which takes alter table/partition
replace flows and stats should be valid after successful replication. Need this
for all non-transactional, transactional and migration cases.
2. Parallel inserts with autogather enabled. Now, we will have events such
that multiple txns open when updating stats event. Also, try to simulate that
one stats update was successful and the other one invalidates it due to
concurrent writes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218867)
Time Spent: 2.5h (was: 2h 20m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218865
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269262756
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java
##
@@ -381,4 +382,11 @@ public void setOwnerName(String ownerName) {
throw new RuntimeException("Invalid table type : " + getDescType());
}
}
+
+ public Long getReplWriteId() {
+if (this.createTblDesc != null) {
+ return this.createTblDesc.getReplWriteId();
Review comment:
This replWriteId is just a place holder for the writeId from the event
message. It need not be in CreateTableDesc. Can be maintained in local
variables and pass around.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218865)
Time Spent: 2h 10m (was: 2h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218860&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218860
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269220469
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2950,21 +2956,33 @@ public Partition createPartition(Table tbl,
Map partSpec) throws
int size = addPartitionDesc.getPartitionCount();
List in =
new ArrayList(size);
-AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf,
tbl, true);
long writeId;
String validWriteIdList;
-if (tableSnapshot != null && tableSnapshot.getWriteId() > 0) {
- writeId = tableSnapshot.getWriteId();
- validWriteIdList = tableSnapshot.getValidWriteIdList();
+
+// In case of replication, get the writeId from the source and use valid
write Id list
+// for replication.
+if (addPartitionDesc.getReplicationSpec() != null &&
+addPartitionDesc.getReplicationSpec().isInReplicationScope() &&
+addPartitionDesc.getPartition(0).getWriteId() > 0) {
+ writeId = addPartitionDesc.getPartition(0).getWriteId();
+ validWriteIdList =
Review comment:
In replication flow, it is fine to use hardcoded ValidWriteIdList as we want
to forcefully set this writeId into table or partition objects. Getting it from
current state might be wrong as we don't update ValidTxnList in conf for repl
created txns.
ValidWriteIdList is just used to check if writeId in metastore objects are
updated by any concurrent inserts. In repl load flow, it is not possible as we
replicate one event at a time or in bootstrap, no 2 threads writes into same
table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218860)
Time Spent: 1.5h (was: 1h 20m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218863
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269169210
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -2130,11 +2144,18 @@ private void create_table_core(final RawStore ms,
final Table tbl,
// If the table has column statistics, update it into the metastore.
This feature is used
// by replication to replicate table level statistics.
- if (tbl.isSetColStats()) {
-// We do not replicate statistics for a transactional table right now
and hence we do not
-// expect a transactional table to have column statistics here. So
passing null
-// validWriteIds is fine for now.
-updateTableColumnStatsInternal(tbl.getColStats(), null,
tbl.getWriteId());
+ if (colStats != null) {
+// On replica craft a valid snapshot out of the writeId in the table.
+long writeId = tbl.getWriteId();
+String validWriteIds = null;
+if (writeId > 0) {
+ ValidWriteIdList vwil =
+ new
ValidReaderWriteIdList(TableName.getDbTable(tbl.getDbName(),
Review comment:
Shall add a comment on why the hardcoded validWriteList is used in this flow
instead of taking current state of txns.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218863)
Time Spent: 1h 50m (was: 1h 40m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218856
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269110947
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -828,6 +828,8 @@ public void alterPartitions(String tblName,
List newParts,
new ArrayList();
try {
AcidUtils.TableSnapshot tableSnapshot = null;
+ // TODO: In case of replication use the writeId and valid write id list
constructed for
Review comment:
Is it done or still TODO?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218856)
Time Spent: 1h (was: 50m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218854&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218854 ] ASF GitHub Bot logged work on HIVE-21109: - Author: ASF GitHub Bot Created on: 26/Mar/19 18:58 Start Date: 26/Mar/19 18:58 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #579: HIVE-21109 : Support stats replication for ACID tables. URL: https://github.com/apache/hive/pull/579#discussion_r269060256 ## File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableDesc.java ## @@ -118,7 +118,8 @@ List notNullConstraints; List defaultConstraints; List checkConstraints; - private ColumnStatistics colStats; + private ColumnStatistics colStats; // For the sake of replication + private long writeId = -1; // For the sake of replication Review comment: Can we re-use the replWriteId variable that we already have? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 218854) Time Spent: 40m (was: 0.5h) > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218853&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218853
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269098036
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
##
@@ -2689,7 +2689,19 @@ private int alterTable(Hive db, AlterTableDesc
alterTbl) throws HiveException {
} else {
// Note: this is necessary for UPDATE_STATISTICS command, that
operates via ADDPROPS (why?).
// For any other updates, we don't want to do txn check on
partitions when altering table.
-boolean isTxn = alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS;
+boolean isTxn = false;
+if (alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS) {
+ // ADDPROPS is used to add repl.last.id during replication. That's
not a transactional
+ // change.
+ Map props = alterTbl.getProps();
+ if (props.size() <= 1 &&
props.get(ReplicationSpec.KEY.CURR_STATE_ID.toString()) != null) {
+isTxn = false;
+ } else {
+isTxn = true;
+ }
+}
+// TODO: Somehow we have to signal alterPartitions that it's part of
replication and
+// should use replication's valid writeid list instead of creating
one.
Review comment:
What do you mean by replication's valid writeid list in this comment? Even
in repl flow, we get validWriteIdList from HMS based on incoming writeId in the
event msg. Are you suggesting to cache this ValidWriteIdList somewhere and use
it instead of invoking HMS API?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218853)
Time Spent: 0.5h (was: 20m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218864
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269223302
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2950,21 +2956,33 @@ public Partition createPartition(Table tbl,
Map partSpec) throws
int size = addPartitionDesc.getPartitionCount();
List in =
new ArrayList(size);
-AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf,
tbl, true);
long writeId;
String validWriteIdList;
-if (tableSnapshot != null && tableSnapshot.getWriteId() > 0) {
- writeId = tableSnapshot.getWriteId();
- validWriteIdList = tableSnapshot.getValidWriteIdList();
+
+// In case of replication, get the writeId from the source and use valid
write Id list
+// for replication.
+if (addPartitionDesc.getReplicationSpec() != null &&
Review comment:
addPartitionDesc.getReplicationSpec() will never be null. Can remove this
check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218864)
Time Spent: 2h (was: 1h 50m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218852&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218852
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269081532
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
##
@@ -2689,7 +2689,19 @@ private int alterTable(Hive db, AlterTableDesc
alterTbl) throws HiveException {
} else {
// Note: this is necessary for UPDATE_STATISTICS command, that
operates via ADDPROPS (why?).
// For any other updates, we don't want to do txn check on
partitions when altering table.
-boolean isTxn = alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS;
+boolean isTxn = false;
+if (alterTbl.getPartSpec() != null && alterTbl.getOp() ==
AlterTableTypes.ADDPROPS) {
+ // ADDPROPS is used to add repl.last.id during replication. That's
not a transactional
+ // change.
+ Map props = alterTbl.getProps();
+ if (props.size() <= 1 &&
props.get(ReplicationSpec.KEY.CURR_STATE_ID.toString()) != null) {
Review comment:
ReplUtils.REPL_CHECKPOINT_KEY is another prop we set it in repl flow which
is not transactional. This check doesn't seems to be clean as in future we
might add more such alters in repl flow. Can we check
replicationSpec.isReplicationScope instead or another flag in AlterTableDesc to
skip this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218852)
Time Spent: 20m (was: 10m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218861
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269161871
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -2130,11 +2144,18 @@ private void create_table_core(final RawStore ms,
final Table tbl,
// If the table has column statistics, update it into the metastore.
This feature is used
// by replication to replicate table level statistics.
- if (tbl.isSetColStats()) {
-// We do not replicate statistics for a transactional table right now
and hence we do not
-// expect a transactional table to have column statistics here. So
passing null
-// validWriteIds is fine for now.
-updateTableColumnStatsInternal(tbl.getColStats(), null,
tbl.getWriteId());
+ if (colStats != null) {
+// On replica craft a valid snapshot out of the writeId in the table.
+long writeId = tbl.getWriteId();
+String validWriteIds = null;
+if (writeId > 0) {
+ ValidWriteIdList vwil =
Review comment:
Shall use meaningful names instead of "vwil".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218861)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218866&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218866
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269257547
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -987,10 +989,14 @@ public void createTable(Table tbl, boolean ifNotExists,
tTbl.setPrivileges(principalPrivs);
}
}
- // Set table snapshot to api.Table to make it persistent.
- TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf, tbl,
true);
- if (tableSnapshot != null) {
-tbl.getTTable().setWriteId(tableSnapshot.getWriteId());
+ // Set table snapshot to api.Table to make it persistent. A
transactional table being
+ // replicated may have a valid write Id copied from the source. Use that
instead of
+ // crafting one on the replica.
+ if (tTbl.getWriteId() <= 0) {
Review comment:
DO_NOT_UPDATE_STATS flag should be set in createTableFlow as well. Or else
in autogather mode at target, it will be updated automatically.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218866)
Time Spent: 2h 20m (was: 2h 10m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218857 ] ASF GitHub Bot logged work on HIVE-21109: - Author: ASF GitHub Bot Created on: 26/Mar/19 18:58 Start Date: 26/Mar/19 18:58 Worklog Time Spent: 10m Work Description: sankarh commented on pull request #579: HIVE-21109 : Support stats replication for ACID tables. URL: https://github.com/apache/hive/pull/579#discussion_r269103325 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/events/filesystem/FSTableEvent.java ## @@ -199,12 +199,15 @@ private AddPartitionDesc partitionDesc(Path fromPath, // Right now, we do not have a way of associating a writeId with statistics for a table // converted to a transactional table if it was non-transactional on the source. So, do not Review comment: Comment needs to be corrected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 218857) > Stats replication for ACID tables. > -- > > Key: HIVE-21109 > URL: https://issues.apache.org/jira/browse/HIVE-21109 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch, > HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch, > HIVE-21109.06.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Transactional tables require a writeID associated with the stats update. This > writeId needs to be in sync with the writeId on the source and hence needs to > be replicated from the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218859
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269154738
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnCommonUtils.java
##
@@ -84,6 +86,73 @@ public static ValidTxnList
createValidReadTxnList(GetOpenTxnsResponse txns, long
return new ValidReadTxnList(exceptions, outAbortedBits, highWaterMark,
minOpenTxnId);
}
+ /**
+ * Transform a {@link
org.apache.hadoop.hive.metastore.api.GetOpenTxnsResponse} to a
+ * {@link org.apache.hadoop.hive.common.ValidTxnList}. This assumes that
the caller intends to
+ * read the files, and thus treats both open and aborted transactions as
invalid.
+ *
+ * This API is used by Hive replication which may have multiple transactions
open at a time.
+ *
+ * @param txns open txn list from the metastore
+ * @param currentTxns Current transactions that the replication has opened.
If any of the
+ *transactions is greater than 0 it will be removed from
the exceptions
+ *list so that the replication sees its own transaction
as valid.
+ * @return a valid txn list.
+ */
+ public static ValidTxnList createValidReadTxnList(GetOpenTxnsResponse txns,
Review comment:
The complete logic of considering all txns opened in a batch by open txn
event as current txns is incorrect.
Multiple txns are opened by repl task only for replicating Hive Streaming
case where we allocate txns batch but use one at a time. Also, we don't update
stats in that case. Even if we update stats, it should refer to one txn as
current txn and rest of the txns are left open.
Shall remove replTxnIds cache in TxnManager as well. All callers shall
create a hardcoded ValidWriteIdList using the writeId received from event msg.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218859)
Time Spent: 1h 20m (was: 1h 10m)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21109) Stats replication for ACID tables.
[
https://issues.apache.org/jira/browse/HIVE-21109?focusedWorklogId=218862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218862
]
ASF GitHub Bot logged work on HIVE-21109:
-
Author: ASF GitHub Bot
Created on: 26/Mar/19 18:58
Start Date: 26/Mar/19 18:58
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #579: HIVE-21109 :
Support stats replication for ACID tables.
URL: https://github.com/apache/hive/pull/579#discussion_r269172695
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -3539,10 +3573,19 @@ public boolean equals(Object obj) {
}
// Update partition column statistics if available
-for (Partition newPart : newParts) {
- if (newPart.isSetColStats()) {
-updatePartitonColStatsInternal(tbl, newPart.getColStats(), null,
newPart.getWriteId());
+int cnt = 0;
+for (ColumnStatistics partColStats: partsColStats) {
+ long writeId = partsWriteIds.get(cnt++);
+ // On replica craft a valid snapshot out of the writeId in the
partition
+ String validWriteIds = null;
+ if (writeId > 0) {
+ValidWriteIdList vwil =
Review comment:
Same as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 218862)
Time Spent: 1h 40m (was: 1.5h)
> Stats replication for ACID tables.
> --
>
> Key: HIVE-21109
> URL: https://issues.apache.org/jira/browse/HIVE-21109
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21109.01.patch, HIVE-21109.02.patch,
> HIVE-21109.03.patch, HIVE-21109.04.patch, HIVE-21109.05.patch,
> HIVE-21109.06.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Transactional tables require a writeID associated with the stats update. This
> writeId needs to be in sync with the writeId on the source and hence needs to
> be replicated from the source.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802073#comment-16802073
]
Simon poortman commented on HIVE-21504:
---
We are trying to use TypeUTILS TO VALIDATE THE DATA TYPE INFORMATION OF A STRING
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21507) Hive swallows NPE if no delegation token found
[
https://issues.apache.org/jira/browse/HIVE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802070#comment-16802070
]
Hive QA commented on HIVE-21507:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} jdbc: The patch generated 1 new + 34 unchanged - 0
fixed = 35 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16693/dev-support/hive-personality.sh
|
| git revision | master / d07cf0d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16693/yetus/diff-checkstyle-jdbc.txt
|
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16693/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive swallows NPE if no delegation token found
> --
>
> Key: HIVE-21507
> URL: https://issues.apache.org/jira/browse/HIVE-21507
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.1.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Labels: usability
> Attachments: HIVE-21507.001.patch, HIVE-21507.002.patch,
> HIVE-21507.003.patch
>
>
> In case if there is no delegation token put into token file, this
> [line|https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L777]
> will cause a NullPointerException which is not handled and the user is not
> notified in any way.
> To cause NPE the use case is to have an Oozie Sqoop import to Hive in a
> kerberized cluster. Oozie puts the delegation token into the token file with
> id: *HIVE_DELEGATION_TOKEN_hiveserver2ClientToken*. So with id *hive* it is
> not working. However, fallback code uses the key which Oozie provides
> [this|https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L784]
> way.
> I suggest to have warning message to user that key with id *hive* cannot be
> used and falling back to get delegation token from the session.
> I am creating the patch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802045#comment-16802045
]
Simon poortman commented on HIVE-21504:
---
Key
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21504) map data type in hive allows complex data type as map key
[
https://issues.apache.org/jira/browse/HIVE-21504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon poortman reassigned HIVE-21504:
-
Assignee: Simon poortman
Activiste the train to my map
> map data type in hive allows complex data type as map key
> -
>
> Key: HIVE-21504
> URL: https://issues.apache.org/jira/browse/HIVE-21504
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Mounica Annaparthi
>Assignee: Simon poortman
>Priority: Major
>
> Hi,
> We are trying to use TypeInfoUtils to validate the data type information of a
> string.
> {code:java}
> TypeInfoUtils.getTypeInfoFromTypeString(inputString).getCategory();
> {code}
> map,string> comes back as data type MAP . I was expecting this to
> give back error as map key should only be a primitive type.
> Could you please check and let us know if this is already known issue or this
> is something that has to be fixed.
> Thanks!
> Mona
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21511) beeline -f report no such file if file is not on local fs
[
https://issues.apache.org/jira/browse/HIVE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802037#comment-16802037
]
Hive QA commented on HIVE-21511:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963740/HIVE-21511.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15840 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16692/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16692/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16692/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963740 - PreCommit-HIVE-Build
> beeline -f report no such file if file is not on local fs
> -
>
> Key: HIVE-21511
> URL: https://issues.apache.org/jira/browse/HIVE-21511
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Blocker
> Labels: patch
> Attachments: HIVE-21511.1.patch
>
> Original Estimate: 0.05h
> Remaining Estimate: 0.05h
>
> I test like this
> HQL=hdfs://hacluster/tmp/ff.hql
> if hadoop fs -test -f ${HQL}
> then
> beeline -f ${HQL}
> fi
> test ${HQL} ok, but beeline report ${HQL} no such file or directory
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Attachment: HIVE-21231.04.patch
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21231) HiveJoinAddNotNullRule support for range predicates
[
https://issues.apache.org/jira/browse/HIVE-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-21231:
--
Status: Patch Available (was: Open)
> HiveJoinAddNotNullRule support for range predicates
> ---
>
> Key: HIVE-21231
> URL: https://issues.apache.org/jira/browse/HIVE-21231
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: newbie
> Attachments: HIVE-21231.01.patch, HIVE-21231.02.patch,
> HIVE-21231.03.patch, HIVE-21231.04.patch
>
>
> For instance, given the following query:
> {code:sql}
> SELECT t0.col0, t0.col1
> FROM
> (
> SELECT col0, col1 FROM tab
> ) AS t0
> INNER JOIN
> (
> SELECT col0, col1 FROM tab
> ) AS t1
> ON t0.col0 < t1.col0 AND t0.col1 > t1.col1
> {code}
> we could still infer that col0 and col1 cannot be null for any of the inputs.
> Currently we do not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
