[jira] [Commented] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410843#comment-16410843 ] Thejas M Nair commented on HIVE-19007: -- +1 > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18885) DbNotificationListener has a deadlock between Java and DB locks
[
https://issues.apache.org/jira/browse/HIVE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18885:
---
Status: Patch Available (was: Open)
> DbNotificationListener has a deadlock between Java and DB locks
> ---
>
> Key: HIVE-18885
> URL: https://issues.apache.org/jira/browse/HIVE-18885
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 2.3.2
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18885.01.branch-2.patch
>
>
> You can see the problem from looking at the code, but it actually created
> severe problems for real life Hive user.
> When {{alter table}} has {{cascade}} option it does the following:
> {code:java}
> msdb.openTransaction()
> ...
> List parts = msdb.getPartitions(dbname, name, -1);
> for (Partition part : parts) {
> List oldCols = part.getSd().getCols();
> part.getSd().setCols(newt.getSd().getCols());
> String oldPartName =
> Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
> updatePartColumnStatsForAlterColumns(msdb, part, oldPartName,
> part.getValues(), oldCols, part);
> msdb.alterPartition(dbname, name, part.getValues(), part);
> }
> {code}
> So it walks all partitions (and this may be huge list) and does some
> non-trivial operations in one single uber-transaction.
> When DbNotificationListener is enabled, it adds an event for each partition,
> all while
> holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is
> happening no other write DDL can proceed. This can sometimes cause DB lock
> timeouts which cause HMS level operation retries which make things even worse.
> In one particular case this pretty much made HMS unusable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18885) DbNotificationListener has a deadlock between Java and DB locks
[
https://issues.apache.org/jira/browse/HIVE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18885:
---
Attachment: HIVE-18885.01.branch-2.patch
> DbNotificationListener has a deadlock between Java and DB locks
> ---
>
> Key: HIVE-18885
> URL: https://issues.apache.org/jira/browse/HIVE-18885
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 2.3.2
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18885.01.branch-2.patch
>
>
> You can see the problem from looking at the code, but it actually created
> severe problems for real life Hive user.
> When {{alter table}} has {{cascade}} option it does the following:
> {code:java}
> msdb.openTransaction()
> ...
> List parts = msdb.getPartitions(dbname, name, -1);
> for (Partition part : parts) {
> List oldCols = part.getSd().getCols();
> part.getSd().setCols(newt.getSd().getCols());
> String oldPartName =
> Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
> updatePartColumnStatsForAlterColumns(msdb, part, oldPartName,
> part.getValues(), oldCols, part);
> msdb.alterPartition(dbname, name, part.getValues(), part);
> }
> {code}
> So it walks all partitions (and this may be huge list) and does some
> non-trivial operations in one single uber-transaction.
> When DbNotificationListener is enabled, it adds an event for each partition,
> all while
> holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is
> happening no other write DDL can proceed. This can sometimes cause DB lock
> timeouts which cause HMS level operation retries which make things even worse.
> In one particular case this pretty much made HMS unusable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18780) Improve schema discovery For Druid Storage Handler
[ https://issues.apache.org/jira/browse/HIVE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410836#comment-16410836 ] Ashutosh Chauhan commented on HIVE-18780: - +1 pending tests. > Improve schema discovery For Druid Storage Handler > -- > > Key: HIVE-18780 > URL: https://issues.apache.org/jira/browse/HIVE-18780 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18780.11.patch, HIVE-18780.12.patch, > HIVE-18780.2.patch, HIVE-18780.4.patch, HIVE-18780.5.patch, > HIVE-18780.6.patch, HIVE-18780.7.patch, HIVE-18780.8.patch, HIVE-18780.patch, > HIVE-18780.patch > > > Currently, Druid Storage adapter issues a Segment metadata Query every time > the query is of type Select or Scan. Not only that but then every input split > (map) will do the same as well since it is using the same Serde, this is very > expensive and put a lot of pressure on the Druid Cluster. The way to fix this > is to add the schema out of the calcite plan instead of serializing the query > itself as part of the Hive query context. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18780) Improve schema discovery For Druid Storage Handler
[
https://issues.apache.org/jira/browse/HIVE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410834#comment-16410834
]
Hive QA commented on HIVE-18780:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
41s{color} | {color:red} root: The patch generated 30 new + 418 unchanged - 41
fixed = 448 total (was 459) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} druid-handler: The patch generated 14 new + 224
unchanged - 41 fixed = 238 total (was 265) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 16 new + 189 unchanged - 0
fixed = 205 total (was 189) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9779/dev-support/hive-personality.sh
|
| git revision | master / 353a9cf |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9779/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9779/yetus/diff-checkstyle-druid-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9779/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9779/yetus/patch-asflicense-problems.txt
|
| modules | C: common . druid-handler itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9779/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve schema discovery For Druid Storage Handler
> --
>
> Key: HIVE-18780
> URL: https://issues.apache.org/jira/browse/HIVE-18780
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18780.11.patch, HIVE-18780.12.patch,
> HIVE-18780.2.patch,
[jira] [Updated] (HIVE-18780) Improve schema discovery For Druid Storage Handler
[ https://issues.apache.org/jira/browse/HIVE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18780: -- Attachment: HIVE-18780.12.patch > Improve schema discovery For Druid Storage Handler > -- > > Key: HIVE-18780 > URL: https://issues.apache.org/jira/browse/HIVE-18780 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18780.11.patch, HIVE-18780.12.patch, > HIVE-18780.2.patch, HIVE-18780.4.patch, HIVE-18780.5.patch, > HIVE-18780.6.patch, HIVE-18780.7.patch, HIVE-18780.8.patch, HIVE-18780.patch, > HIVE-18780.patch > > > Currently, Druid Storage adapter issues a Segment metadata Query every time > the query is of type Select or Scan. Not only that but then every input split > (map) will do the same as well since it is using the same Serde, this is very > expensive and put a lot of pressure on the Druid Cluster. The way to fix this > is to add the schema out of the calcite plan instead of serializing the query > itself as part of the Hive query context. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19019) Vectorization and Parquet: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410767#comment-16410767
]
Hive QA commented on HIVE-19019:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915708/HIVE-19019.02.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 332 failed/errored test(s), 13093 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Assigned] (HIVE-18996) SubString Druid convertor assuming that index is always constant literal value
[
https://issues.apache.org/jira/browse/HIVE-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra reassigned HIVE-18996:
-
Assignee: slim bouguerra
> SubString Druid convertor assuming that index is always constant literal value
> --
>
> Key: HIVE-18996
> URL: https://issues.apache.org/jira/browse/HIVE-18996
> Project: Hive
> Issue Type: Bug
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Minor
>
> Query like the following
> {code}
> SELECT substring(namespace, CAST(deleted AS INT), 4)
> FROM druid_table_1;
> {code}
> will fail with
> {code}
> java.lang.AssertionError: not a literal: $13
> at org.apache.calcite.rex.RexLiteral.findValue(RexLiteral.java:963)
> at org.apache.calcite.rex.RexLiteral.findValue(RexLiteral.java:955)
> at org.apache.calcite.rex.RexLiteral.intValue(RexLiteral.java:938)
> at
> org.apache.calcite.adapter.druid.SubstringOperatorConversion.toDruidExpression(SubstringOperatorConversion.java:46)
> at
> org.apache.calcite.adapter.druid.DruidExpressions.toDruidExpression(DruidExpressions.java:120)
> at
> org.apache.calcite.adapter.druid.DruidQuery.computeProjectAsScan(DruidQuery.java:746)
> at
> org.apache.calcite.adapter.druid.DruidRules$DruidProjectRule.onMatch(DruidRules.java:308)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:317)
> {code}
> because is assuming that index is always a constant literal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19019) Vectorization and Parquet: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410730#comment-16410730
]
Hive QA commented on HIVE-19019:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 5 new + 117 unchanged - 0
fixed = 122 total (was 117) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9778/dev-support/hive-personality.sh
|
| git revision | master / 353a9cf |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9778/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9778/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9778/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9778/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization and Parquet: When vectorized, orc_merge_incompat_schema.q
> throws HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-18952) Tez session disconnect and reconnect on HS2 HA failover
[ https://issues.apache.org/jira/browse/HIVE-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410721#comment-16410721 ] Prasanth Jayachandran commented on HIVE-18952: -- +1, pending tests. > Tez session disconnect and reconnect on HS2 HA failover > --- > > Key: HIVE-18952 > URL: https://issues.apache.org/jira/browse/HIVE-18952 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18952.01.patch, HIVE-18952.patch > > > Now that TEZ-3892 is committed, HIVE-18281 can make use of tez session > disconnect and reconnect on HA failover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410719#comment-16410719
]
Sahil Takiar commented on HIVE-18831:
-
[~lirui] addressed your comments. Took a slightly different approach to how
exceptions are handling, details are in the RB. TL;DR is that I think we can
serialize {{Throwable}} objects from the {{RemoteDriver}} without having to
shove them into a {{String}}. Let me know what you think.
> Differentiate errors that are thrown by Spark tasks
> ---
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch,
> HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we
> don't differentiate between errors from HS2 / RSC vs. errors thrown by
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its
> difficult to know what part of the excution threw the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18831: Attachment: HIVE-18831.6.patch > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, > HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18831: Attachment: (was: HIVE-18831.5.patch) > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, > HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18831: Attachment: HIVE-18831.5.patch > Differentiate errors that are thrown by Spark tasks > --- > > Key: HIVE-18831 > URL: https://issues.apache.org/jira/browse/HIVE-18831 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, > HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch > > > We propagate exceptions from Spark task failures to the client well, but we > don't differentiate between errors from HS2 / RSC vs. errors thrown by > individual tasks. > Main motivation is that when the client sees a propagated Spark exception its > difficult to know what part of the excution threw the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18712) Design HMS Api v2
[ https://issues.apache.org/jira/browse/HIVE-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410707#comment-16410707 ] Alexander Kolbasov commented on HIVE-18712: --- Would using TLS encryption be an issue? For example, if we use Kerberos for authentication and TLS for encryption? > Design HMS Api v2 > - > > Key: HIVE-18712 > URL: https://issues.apache.org/jira/browse/HIVE-18712 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > This is an umbrella Jira covering the design of Hive Metastore API v2. > It is supposed to be a placeholder for discussion and design documents. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18712) Design HMS Api v2
[ https://issues.apache.org/jira/browse/HIVE-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410705#comment-16410705 ] Alexander Kolbasov commented on HIVE-18712: --- I am not an expert on this, but I know of at least one product that uses gRPC and Kerberos and it works. I'll try to get more details. > Design HMS Api v2 > - > > Key: HIVE-18712 > URL: https://issues.apache.org/jira/browse/HIVE-18712 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > This is an umbrella Jira covering the design of Hive Metastore API v2. > It is supposed to be a placeholder for discussion and design documents. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410702#comment-16410702 ] Prasanth Jayachandran commented on HIVE-19033: -- There is no security yet. If this is required to be admin only feature and work only for secure clusters then I will probably do something like HIVE-18982. There might be a need for this to be run on unsecure clusters as well hence leaving it as such. We can always enable hadoop.security.instrumentation.requires.admin to restrict access to this endpoint (default is false though) > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18971) add HS2 WM metrics for use in Grafana and such
[ https://issues.apache.org/jira/browse/HIVE-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18971: Status: Patch Available (was: Open) Still need to test on cluster. I think I'll also add codahale counters, just prefixed with pool name, so they'd display in metrics dump. Looks like that's the approach taken for e.g. APIs, I see metrics like " "name" : "metrics:name=active_calls_api_TezGetSession"," Not sure if we'll be able to do anything intelligent with Grafana with that and custom pools :( Hopefully regular metrics work in parallel with codahale. cc [~prasanth_j] does this patch make sense? > add HS2 WM metrics for use in Grafana and such > -- > > Key: HIVE-18971 > URL: https://issues.apache.org/jira/browse/HIVE-18971 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18971.patch > > > HS2 should have metrics added per pool, tagged accordingly. Not clear if HS2 > even sets up metrics right now... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410694#comment-16410694
]
Hive QA commented on HIVE-15995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915705/HIVE-15995.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 329 failed/errored test(s), 13486 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410692#comment-16410692 ] Prasanth Jayachandran commented on HIVE-19033: -- Initial patch. Tested the patch only locally one single node. Will try test it on a cluster and update the patch in case of issues. [~sershe]/[~gopalv] can someone please review this patch? > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19033: - Status: Patch Available (was: Open) > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19033: - Attachment: HIVE-19033.1.patch > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-19033: > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18971) add HS2 WM metrics for use in Grafana and such
[ https://issues.apache.org/jira/browse/HIVE-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410679#comment-16410679 ] Sergey Shelukhin commented on HIVE-18971: - Initial patch. I was hoping to propagate some stuff from AM thru progress but it looks like that would require Tez changes. Also no idea how to output many sources with codahale without reinitializing the whole metrics system so I'm just using Hadoop metrics for now. > add HS2 WM metrics for use in Grafana and such > -- > > Key: HIVE-18971 > URL: https://issues.apache.org/jira/browse/HIVE-18971 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18971.patch > > > HS2 should have metrics added per pool, tagged accordingly. Not clear if HS2 > even sets up metrics right now... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18971) add HS2 WM metrics for use in Grafana and such
[ https://issues.apache.org/jira/browse/HIVE-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18971: Attachment: HIVE-18971.patch > add HS2 WM metrics for use in Grafana and such > -- > > Key: HIVE-18971 > URL: https://issues.apache.org/jira/browse/HIVE-18971 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18971.patch > > > HS2 should have metrics added per pool, tagged accordingly. Not clear if HS2 > even sets up metrics right now... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410668#comment-16410668
]
Hive QA commented on HIVE-15995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
50s{color} | {color:red} root: The patch generated 5 new + 808 unchanged - 0
fixed = 813 total (was 808) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 5 new + 808 unchanged - 0
fixed = 813 total (was 808) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9777/dev-support/hive-personality.sh
|
| git revision | master / 353a9cf |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus/patch-asflicense-problems.txt
|
| modules | C: . ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9777/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch,
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch,
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can
> alter the 'avro.schema.url' property to update table schema to the next
> version. Updating properties however doesn't affect column list stored in
> metastore DB so the table is not in the newest version when returned from
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement
[jira] [Updated] (HIVE-18909) Metrics for results cache
[ https://issues.apache.org/jira/browse/HIVE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18909: -- Attachment: HIVE-18909.2.patch > Metrics for results cache > - > > Key: HIVE-18909 > URL: https://issues.apache.org/jira/browse/HIVE-18909 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Labels: Metrics > Attachments: HIVE-18909.1.patch, HIVE-18909.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18433) Upgrade version of com.fasterxml.jackson
[ https://issues.apache.org/jira/browse/HIVE-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18433: --- Attachment: HIVE-18433.7.patch > Upgrade version of com.fasterxml.jackson > > > Key: HIVE-18433 > URL: https://issues.apache.org/jira/browse/HIVE-18433 > Project: Hive > Issue Type: Task >Reporter: Sahil Takiar >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18433.1.patch, HIVE-18433.2.patch, > HIVE-18433.3.patch, HIVE-18433.4.patch, HIVE-18433.5.patch, > HIVE-18433.6.patch, HIVE-18433.7.patch > > > Let's upgrade to version 2.9.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410648#comment-16410648 ] Eugene Koifman commented on HIVE-18739: --- you can't use a temp table for this - it doesn't support partitioned tables > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, > HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, > HIVE-18739.12.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18780) Improve schema discovery For Druid Storage Handler
[ https://issues.apache.org/jira/browse/HIVE-18780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18780: -- Attachment: HIVE-18780.11.patch > Improve schema discovery For Druid Storage Handler > -- > > Key: HIVE-18780 > URL: https://issues.apache.org/jira/browse/HIVE-18780 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18780.11.patch, HIVE-18780.2.patch, > HIVE-18780.4.patch, HIVE-18780.5.patch, HIVE-18780.6.patch, > HIVE-18780.7.patch, HIVE-18780.8.patch, HIVE-18780.patch, HIVE-18780.patch > > > Currently, Druid Storage adapter issues a Segment metadata Query every time > the query is of type Select or Scan. Not only that but then every input split > (map) will do the same as well since it is using the same Serde, this is very > expensive and put a lot of pressure on the Druid Cluster. The way to fix this > is to add the schema out of the calcite plan instead of serializing the query > itself as part of the Hive query context. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410633#comment-16410633 ] Sergey Shelukhin edited comment on HIVE-18739 at 3/23/18 12:47 AM: --- Code looks ok to me modulo some whitespace, todos, etc. However I'm -0 on the approach. First there are potential security implications with shared database. I think the table at least has to be session-temporary. I'm not sure how safe those are in the first place, but right now we create a globally visible table in a shared DB, potentially giving users with access to any database and export access to data from every other database (by grabbing it during export). It's protected only by the obscurity of the table name. I'm -1 on that. Overall it seems like using full compaction, so we could benefit every other reader of the table (and future export, see side note) would still be better, and also not multiply entities and have simpler code without compiling a query, having users set up a special DB, etc. No matter how compaction evolves it should still be possible to run full compaction by including all deltas; it will be a flag and minimal code difference from future improved compaction, during delta selection. But I'm not going to stop this patch as long as security issue is addressed :) If continuing with this approach, I wonder if we could, instead of relying on custom db, create a new type of privilege check, and allow users to create the export table in the source db even if they don't have permissions to create normal tables there? Or we can use session scoped temp table. Or materialized view? cc [~ashutoshc] [~thejas] to review As a side note, one optimization here is that if table is already fully compacted, we could just take the base and export it directly without using insert. was (Author: sershe): Code looks ok to me modulo some whitespace, todos, etc. However I'm -0 on the approach. First there are potential security implications with shared database. I think it the minimum, the table has to be session-temporary. I'm not sure how safe those are in the first place, but right now we create a globally visible table in a shared DB, potentially giving users with access to any database and export access to data from every other database (by grabbing it during export). It's protected only by the obscurity of the table name. I'm -1 on that. Overall it seems like using full compaction, so we could benefit every other reader of the table (and future export, see side note) would still be better, and also not multiply entities and have simpler code without compiling a query, having users set up a special DB, etc. No matter how compaction evolves it should still be possible to run full compaction by including all deltas; it will be a flag and minimal code difference from future improved compaction, during delta selection. But I'm not going to stop this patch as long as security issue is addressed :) If continuing with this approach, I wonder if we could, instead of relying on custom db, create a new type of privilege check, and allow users to create the export table in the source db even if they don't have permissions to create normal tables there? Or we can use session scoped temp table. Or materialized view? cc [~ashutoshc] [~thejas] to review As a side note, one optimization here is that if table is already fully compacted, we could just take the base and export it directly without using insert. > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, > HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, > HIVE-18739.12.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18739) Add support for Export from Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410633#comment-16410633 ] Sergey Shelukhin commented on HIVE-18739: - Code looks ok to me modulo some whitespace, todos, etc. However I'm -0 on the approach. First there are potential security implications with shared database. I think it the minimum, the table has to be session-temporary. I'm not sure how safe those are in the first place, but right now we create a globally visible table in a shared DB, potentially giving users with access to any database and export access to data from every other database (by grabbing it during export). It's protected only by the obscurity of the table name. I'm -1 on that. Overall it seems like using full compaction, so we could benefit every other reader of the table (and future export, see side note) would still be better, and also not multiply entities and have simpler code without compiling a query, having users set up a special DB, etc. No matter how compaction evolves it should still be possible to run full compaction by including all deltas; it will be a flag and minimal code difference from future improved compaction, during delta selection. But I'm not going to stop this patch as long as security issue is addressed :) If continuing with this approach, I wonder if we could, instead of relying on custom db, create a new type of privilege check, and allow users to create the export table in the source db even if they don't have permissions to create normal tables there? Or we can use session scoped temp table. Or materialized view? cc [~ashutoshc] [~thejas] to review As a side note, one optimization here is that if table is already fully compacted, we could just take the base and export it directly without using insert. > Add support for Export from Acid table > -- > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, > HIVE-18739.04.patch, HIVE-18739.06.patch, HIVE-18739.08.patch, > HIVE-18739.09.patch, HIVE-18739.10.patch, HIVE-18739.11.patch, > HIVE-18739.12.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410623#comment-16410623
]
Hive QA commented on HIVE-18140:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915527/HIVE-18140.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9776/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9776/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9776/
Messages:
{noformat}
This message was trimmed, see log for full details
+ git fetch origin
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
+ git reset --hard HEAD
HEAD is now at 353a9cf HIVE-19013. Fix various storage-api build issues. (Owen
O'Malley, reviewed by Prasanth Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 353a9cf HIVE-19013. Fix various storage-api build issues. (Owen
O'Malley, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-23 00:29:23.410
+ rm -rf ../yetus_PreCommit-HIVE-Build-9776
+ mkdir ../yetus_PreCommit-HIVE-Build-9776
+ git gc
+ sleep 1s
+ git gc
+ sleep 1s
+ git gc
+ sleep 1s
+ git gc
+ sleep 1s
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9776
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9776/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: ql/src/test/results/clientpositive/insert1_overwrite_partitions.q.out:
does not exist in index
error: qtest/pom.xml: does not exist in index
error:
src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/plan/Statistics.java: does not exist
in index
error: src/java/org/apache/hadoop/hive/ql/stats/Partish.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java: does not exist
in index
error: src/test/queries/clientpositive/stats8.q: does not exist in index
error: src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out: does not
exist in index
error: src/test/results/clientpositive/analyze_table_null_partition.q.out: does
not exist in index
error: src/test/results/clientpositive/annotate_stats_part.q.out: does not
exist in index
error: src/test/results/clientpositive/autoColumnStats_2.q.out: does not exist
in index
error: src/test/results/clientpositive/auto_sortmerge_join_1.q.out: does not
exist in index
error: src/test/results/clientpositive/auto_sortmerge_join_11.q.out: does not
exist in index
error: src/test/results/clientpositive/auto_sortmerge_join_12.q.out: does not
exist in index
error: src/test/results/clientpositive/auto_sortmerge_join_2.q.out: does not
exist in index
error: src/test/results/clientpositive/auto_sortmerge_join_3.q.out: does not
exist in index
error: src/test/results/clientpositive/auto_sortmerge_join_4.q.out: does not
exist in index
error: src/test/results/clientpositive/auto_sortmerge_join_7.q.out: does not
exist in index
error: src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out: does not
exist in index
error: src/test/results/clientpositive/bucket_map_join_spark1.q.out: does not
exist in index
error: src/test/results/clientpositive/bucket_map_join_spark2.q.out: does not
exist in index
error: src/test/results/clientpositive/bucket_map_join_spark3.q.out: does not
exist in index
error: src/test/results/clientpositive/bucketcontext_1.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketcontext_2.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketcontext_3.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketcontext_4.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketcontext_6.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketcontext_7.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketcontext_8.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketmapjoin10.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketmapjoin11.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketmapjoin12.q.out: does not exist in
index
error: src/test/results/clientpositive/bucketmapjoin5.q.out: does not exist in
index
error:
[jira] [Commented] (HIVE-17098) Race condition in Hbase tables
[
https://issues.apache.org/jira/browse/HIVE-17098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410612#comment-16410612
]
Hive QA commented on HIVE-17098:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12877546/HIVE-17098.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 329 failed/errored test(s), 13088 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Updated] (HIVE-19029) Load Data should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19029:
--
Attachment: HIVE-19029.01.patch
> Load Data should prevent loading acid files
> ---
>
> Key: HIVE-19029
> URL: https://issues.apache.org/jira/browse/HIVE-19029
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19029.01.patch
>
>
> {{Load Data into T}} where T is acid should check to make sure input files
> were not copied from another Acid table, i.e. make sure the files don't have
> Acid metadata columns.
> AcidUtils.MetaData.isRawFormat()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19029) Load Data should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19029:
--
Status: Patch Available (was: Open)
> Load Data should prevent loading acid files
> ---
>
> Key: HIVE-19029
> URL: https://issues.apache.org/jira/browse/HIVE-19029
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19029.01.patch
>
>
> {{Load Data into T}} where T is acid should check to make sure input files
> were not copied from another Acid table, i.e. make sure the files don't have
> Acid metadata columns.
> AcidUtils.MetaData.isRawFormat()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18855) Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
[ https://issues.apache.org/jira/browse/HIVE-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410557#comment-16410557 ] Sergey Shelukhin commented on HIVE-18855: - +1 > Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] > --- > > Key: HIVE-18855 > URL: https://issues.apache.org/jira/browse/HIVE-18855 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18855.1.patch > > > Looks like this test has been broken for a while. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18855) Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
[ https://issues.apache.org/jira/browse/HIVE-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410552#comment-16410552 ] Jason Dere commented on HIVE-18855: --- [~sershe] can you review, small fix to set a conf value in the case that the results cache is used. > Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] > --- > > Key: HIVE-18855 > URL: https://issues.apache.org/jira/browse/HIVE-18855 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18855.1.patch > > > Looks like this test has been broken for a while. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18855) Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
[ https://issues.apache.org/jira/browse/HIVE-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410548#comment-16410548 ] Jason Dere edited comment on HIVE-18855 at 3/22/18 11:19 PM: - Test is failing when trying to retrieve the cached results from a UNION query, because UNION queries with Tez execution engine generate nested result directories for each part of the UNION. INPUT_DIR_RECURSIVE needs to be set when listing files from the result directory. Looks like this broke after HIVE-18238 - previously the HiveConf must have been set with INPUT_DIR_RECURSIVE from the first query running in Tez mode, and this setting stayed for all queries run by the test. was (Author: jdere): Looks like this broke after HIVE-18238. Test is failing when trying to retrieve the cached results from a UNION query, because UNION queries with Tez execution engine generate nested result directories for each part of the UNION. INPUT_DIR_RECURSIVE needs to be set when listing files from the result directory. > Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] > --- > > Key: HIVE-18855 > URL: https://issues.apache.org/jira/browse/HIVE-18855 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18855.1.patch > > > Looks like this test has been broken for a while. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18855) Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
[ https://issues.apache.org/jira/browse/HIVE-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18855: -- Attachment: HIVE-18855.1.patch > Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] > --- > > Key: HIVE-18855 > URL: https://issues.apache.org/jira/browse/HIVE-18855 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18855.1.patch > > > Looks like this test has been broken for a while. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18855) Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
[ https://issues.apache.org/jira/browse/HIVE-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18855: -- Status: Patch Available (was: Open) > Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] > --- > > Key: HIVE-18855 > URL: https://issues.apache.org/jira/browse/HIVE-18855 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18855.1.patch > > > Looks like this test has been broken for a while. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18855) Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
[ https://issues.apache.org/jira/browse/HIVE-18855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410548#comment-16410548 ] Jason Dere commented on HIVE-18855: --- Looks like this broke after HIVE-18238. Test is failing when trying to retrieve the cached results from a UNION query, because UNION queries with Tez execution engine generate nested result directories for each part of the UNION. INPUT_DIR_RECURSIVE needs to be set when listing files from the result directory. > Fix unit test TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] > --- > > Key: HIVE-18855 > URL: https://issues.apache.org/jira/browse/HIVE-18855 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > > Looks like this test has been broken for a while. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19032: Status: Patch Available (was: Open) > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19032.01.patch > > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410539#comment-16410539 ] Matt McCline commented on HIVE-19032: - Includes add vectorization disable to the top of regular GROUP BY Q files, etc. > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19032.01.patch > > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17098) Race condition in Hbase tables
[
https://issues.apache.org/jira/browse/HIVE-17098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410540#comment-16410540
]
Hive QA commented on HIVE-17098:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-9775/patches/PreCommit-HIVE-Build-9775.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9775/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Race condition in Hbase tables
> --
>
> Key: HIVE-17098
> URL: https://issues.apache.org/jira/browse/HIVE-17098
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 2.1.1
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Critical
> Attachments: HIVE-17098.1.patch
>
>
> These steps simulate our customer production env.
> *STEP 1. Create test tables*
> {code}
> CREATE TABLE for_loading(
> key int,
> value string,
> age int,
> salary decimal (10,2)
> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
> {code}
> Table {{test_1}} belongs to user {{testuser1}}.
> {code}
> CREATE TABLE test_1(
> key int,
> value string,
> age int,
> salary decimal (10,2)
> )
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key, cf1:value, cf1:age, cf1:salary',
> 'serialization.format'='1')
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'hbase.table.name'='test_1',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1495769316');
> {code}
> Table {{test_2}} belongs to user {{testuser2}}.
> {code}
> CREATE TABLE test_2(
> key int,
> value string,
> age int,
> salary decimal (10,2)
> )
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping'=':key, cf1:value, cf1:age, cf1:salary',
> 'serialization.format'='1')
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'hbase.table.name'='test_2',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1495769316');
> {code}
> *STEP 2. Create test data*
> {code}
> import java.io.IOException;
> import java.math.BigDecimal;
> import java.nio.charset.Charset;
> import java.nio.file.Files;
> import java.nio.file.Path;
> import java.nio.file.Paths;
> import java.nio.file.StandardOpenOption;
> import java.util.ArrayList;
> import java.util.Arrays;
> import java.util.List;
> import java.util.Random;
> import static java.lang.String.format;
> public class Generator {
> private static List lines = new ArrayList<>();
> private static List name = Arrays.asList("Brian", "John",
> "Rodger", "Max", "Freddie", "Albert", "Fedor", "Lev", "Niccolo");
> private static List salary = new ArrayList<>();
> public static void main(String[] args) {
> generateData(Integer.parseInt(args[0]), args[1]);
> }
> public static void generateData(int rowNumber, String file) {
> double maxValue = 2.55;
> double minValue = 1000.03;
> Random random = new Random();
> for (int i = 1; i <= rowNumber; i++) {
> lines.add(
> i + "," +
> name.get(random.nextInt(name.size())) + "," +
> (random.nextInt(62) + 18) + "," +
> format("%.2f", (minValue + (maxValue - minValue) *
> random.nextDouble(;
> }
> Path path = Paths.get(file);
> try {
> Files.write(path, lines, Charset.forName("UTF-8"),
> StandardOpenOption.APPEND);
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> {code}
> {code}
> javac Generator.java
> java Generator 300 dataset.csv
> hadoop fs -put dataset.csv /
> {code}
> *STEP 3. Upload test data*
> {code}
> load data local inpath '/home/myuser/dataset.csv' into table for_loading;
> {code}
> {code}
> from for_loading
> insert into table test_1
> select key,value,age,salary;
> {code}
> {code}
> from for_loading
> insert into table test_2
> select key,value,age,salary;
> {code}
> *STEP 4. Run test queries*
> Run in 5 parallel terminals for table {{test_1}}
>
[jira] [Updated] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19032: Attachment: HIVE-19032.01.patch > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19032.01.patch > > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-19032: --- > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18926) Imporve operator-tree matching
[
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410516#comment-16410516
]
Hive QA commented on HIVE-18926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915507/HIVE-18926.05.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 333 failed/errored test(s), 13091 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Resolved] (HIVE-19013) Fix some minor build issues in storage-api
[ https://issues.apache.org/jira/browse/HIVE-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HIVE-19013. -- Resolution: Fixed Fix Version/s: storage-2.5.0 > Fix some minor build issues in storage-api > -- > > Key: HIVE-19013 > URL: https://issues.apache.org/jira/browse/HIVE-19013 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: storage-2.5.0 > > > Currently, the storage-api tests complain that there isn't a log4j2.xml and > the javadoc fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19031) Mark duplicate configs in HiveConf as deprecated
[ https://issues.apache.org/jira/browse/HIVE-19031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19031: -- Attachment: HIVE-19031.patch > Mark duplicate configs in HiveConf as deprecated > > > Key: HIVE-19031 > URL: https://issues.apache.org/jira/browse/HIVE-19031 > Project: Hive > Issue Type: Sub-task > Components: Configuration, Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-19031.patch > > > There are a number of configuration values that were copied from HiveConf to > MetastoreConf. They have been left in HiveConf for backwards compatibility. > But they need to be marked as deprecated so that users know to use the new > values in MetastoreConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19031) Mark duplicate configs in HiveConf as deprecated
[ https://issues.apache.org/jira/browse/HIVE-19031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410515#comment-16410515 ] Alan Gates commented on HIVE-19031: --- PR for this patch: https://github.com/apache/hive/pull/325 > Mark duplicate configs in HiveConf as deprecated > > > Key: HIVE-19031 > URL: https://issues.apache.org/jira/browse/HIVE-19031 > Project: Hive > Issue Type: Sub-task > Components: Configuration, Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HIVE-19031.patch > > > There are a number of configuration values that were copied from HiveConf to > MetastoreConf. They have been left in HiveConf for backwards compatibility. > But they need to be marked as deprecated so that users know to use the new > values in MetastoreConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19013) Fix some minor build issues in storage-api
[ https://issues.apache.org/jira/browse/HIVE-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410513#comment-16410513 ] ASF GitHub Bot commented on HIVE-19013: --- Github user omalley closed the pull request at: https://github.com/apache/hive/pull/323 > Fix some minor build issues in storage-api > -- > > Key: HIVE-19013 > URL: https://issues.apache.org/jira/browse/HIVE-19013 > Project: Hive > Issue Type: Bug > Components: storage-api >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > > Currently, the storage-api tests complain that there isn't a log4j2.xml and > the javadoc fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19031) Mark duplicate configs in HiveConf as deprecated
[ https://issues.apache.org/jira/browse/HIVE-19031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19031: -- Status: Patch Available (was: Open) This patch marks entries in HiveConf as deprecated. A few of the entries had been added since 2.3.2. These I removed instead, as there's no need to deprecate something that was never released. I also refactored MetastoreConf a bit to pull the validators out into their own files and renamed a few config vars to better match the others. > Mark duplicate configs in HiveConf as deprecated > > > Key: HIVE-19031 > URL: https://issues.apache.org/jira/browse/HIVE-19031 > Project: Hive > Issue Type: Sub-task > Components: Configuration, Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > There are a number of configuration values that were copied from HiveConf to > MetastoreConf. They have been left in HiveConf for backwards compatibility. > But they need to be marked as deprecated so that users know to use the new > values in MetastoreConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19031) Mark duplicate configs in HiveConf as deprecated
[ https://issues.apache.org/jira/browse/HIVE-19031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-19031: - > Mark duplicate configs in HiveConf as deprecated > > > Key: HIVE-19031 > URL: https://issues.apache.org/jira/browse/HIVE-19031 > Project: Hive > Issue Type: Sub-task > Components: Configuration, Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > There are a number of configuration values that were copied from HiveConf to > MetastoreConf. They have been left in HiveConf for backwards compatibility. > But they need to be marked as deprecated so that users know to use the new > values in MetastoreConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[ https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18910: -- Attachment: HIVE-18910.11.patch > Migrate to Murmur hash for shuffle and bucketing > > > Key: HIVE-18910 > URL: https://issues.apache.org/jira/browse/HIVE-18910 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, > HIVE-18910.11.patch, HIVE-18910.2.patch, HIVE-18910.3.patch, > HIVE-18910.4.patch, HIVE-18910.5.patch, HIVE-18910.6.patch, > HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch > > > Hive uses JAVA hash which is not as good as murmur for better distribution > and efficiency in bucketing a table. > Migrate to murmur hash but still keep backward compatibility for existing > users so that they dont have to reload the existing tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19030) Update Wiki with new rules for Load Data
[ https://issues.apache.org/jira/browse/HIVE-19030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19030: - > Update Wiki with new rules for Load Data > > > Key: HIVE-19030 > URL: https://issues.apache.org/jira/browse/HIVE-19030 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Deepak Jaiswal >Priority: Major > > [~djaiswal] could you please update > https://cwiki.apache.org/confluence/display/hive/languagemanual+dml#LanguageManualDML-Loadingfilesintotables > with latest rules based on HIVE-18125 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19029) Load Data should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19029:
--
Issue Type: Sub-task (was: Bug)
Parent: HIVE-17361
> Load Data should prevent loading acid files
> ---
>
> Key: HIVE-19029
> URL: https://issues.apache.org/jira/browse/HIVE-19029
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{Load Data into T}} where T is acid should check to make sure input files
> were not copied from another Acid table, i.e. make sure the files don't have
> Acid metadata columns.
> AcidUtils.MetaData.isRawFormat()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19029) Load Data should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-19029:
-
> Load Data should prevent loading acid files
> ---
>
> Key: HIVE-19029
> URL: https://issues.apache.org/jira/browse/HIVE-19029
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{Load Data into T}} where T is acid should check to make sure input files
> were not copied from another Acid table, i.e. make sure the files don't have
> Acid metadata columns.
> AcidUtils.MetaData.isRawFormat()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410347#comment-16410347
]
Marcelo Vanzin commented on HIVE-18533:
---
That looks like a bug in Spark's {{LauncherBackend}}. It shouldn't be trying to
write to the connection socket if it's been disconnected. The code already
tracks that state, just fails to use it in the write methods (such as
{{setState}}).
> Add option to use InProcessLauncher to submit spark jobs
>
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch,
> HIVE-18533.3.patch, HIVE-18533.4.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a
> HoS session + debuggability (no need launch a separate process to run a Spark
> app).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18926) Imporve operator-tree matching
[
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410345#comment-16410345
]
Hive QA commented on HIVE-18926:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 2 new + 374 unchanged - 15
fixed = 376 total (was 389) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9774/dev-support/hive-personality.sh
|
| git revision | master / 560bca0 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9774/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9774/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9774/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Imporve operator-tree matching
> --
>
> Key: HIVE-18926
> URL: https://issues.apache.org/jira/browse/HIVE-18926
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18926.01.patch, HIVE-18926.02.patch,
> HIVE-18926.03.patch, HIVE-18926.04.patch, HIVE-18926.05.patch
>
>
> currently joins are not matched
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18982) Provide a CLI option to manually trigger failover
[ https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410337#comment-16410337 ] Prasanth Jayachandran commented on HIVE-18982: -- minor update: Removes the response modification if "isInstrumentationAccessAllowed()" call fails as the methods in itself sends error back in the response. So no use adding more to it which won't reach the client anyways. > Provide a CLI option to manually trigger failover > - > > Key: HIVE-18982 > URL: https://issues.apache.org/jira/browse/HIVE-18982 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18982.1.patch, HIVE-18982.2.patch, > HIVE-18982.3.patch, HIVE-18982.4.patch, HIVE-18982.5.patch > > > HIVE-18281 added active-passive HA. There might be a administrative need to > trigger a manual failover of HS2 Active server. Add command line tool to view > list of all HS2 instances and trigger manual failover (only under force > mode). The clients currently connected to active HS2 will be closed. In > future, more options to existing clients connections can be handled via > configs/options (like wait until timeout, wait until current sessions are > closed etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18982) Provide a CLI option to manually trigger failover
[ https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18982: - Attachment: HIVE-18982.5.patch > Provide a CLI option to manually trigger failover > - > > Key: HIVE-18982 > URL: https://issues.apache.org/jira/browse/HIVE-18982 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18982.1.patch, HIVE-18982.2.patch, > HIVE-18982.3.patch, HIVE-18982.4.patch, HIVE-18982.5.patch > > > HIVE-18281 added active-passive HA. There might be a administrative need to > trigger a manual failover of HS2 Active server. Add command line tool to view > list of all HS2 instances and trigger manual failover (only under force > mode). The clients currently connected to active HS2 will be closed. In > future, more options to existing clients connections can be handled via > configs/options (like wait until timeout, wait until current sessions are > closed etc.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Status: Patch Available (was: Open) > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410317#comment-16410317 ] Sankar Hariappan commented on HIVE-18747: - Attached 02.patch with * Implementation as per description. * getValidWriteIdsForTable undergone changes to get writeIdHwm from NEXT_WRITE_ID table if TXN_TO_WRITE_ID table is empty. This can happen if all txns are committed and cleaner removes all entries from TXN_TO_WRITE_ID. > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19028) Insert values query gives Failed to retrieve table statistics
[ https://issues.apache.org/jira/browse/HIVE-19028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19028: -- > Insert values query gives Failed to retrieve table statistics > - > > Key: HIVE-19028 > URL: https://issues.apache.org/jira/browse/HIVE-19028 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Thejas M Nair >Assignee: Jesus Camacho Rodriguez >Priority: Major > > INSERT INTO `VERSION` VALUES (1, '3.0.0', 'Hive release version 3.0.0') gives > error like - > 2018-03-22T17:09:42,680 ERROR [946f78ac-24e9-493b-b877-336d3006a9b5 > HiveServer2-Handler-Pool: Thread-87]: stats.StatsUtils (:()) - Failed to > retrieve table statistics: > org.apache.hadoop.hive.ql.metadata.HiveException: > NoSuchObjectException(message:Specified database/table does not exist : > _dummy_database._dummy_table) > The query proceeds, but this can be confusing to users, support team etc > Full stack is here - > https://gist.github.com/thejasmn/7e33ad38e7bc552e3a659adc041ba200 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Attachment: HIVE-18747.02.patch > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19027) Make materializations invalidation cache work with multiple active remote metastores
[ https://issues.apache.org/jira/browse/HIVE-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-19027: -- > Make materializations invalidation cache work with multiple active remote > metastores > > > Key: HIVE-19027 > URL: https://issues.apache.org/jira/browse/HIVE-19027 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > The main points: > - Only MVs stored in transactional tables can have a time window value of 0. > Those are the only MVs that can be guaranteed to not be outdated when a query > is executed, if we use custom storage handlers to store the materialized > view, we cannot make any promises. > - For MVs that +cannot be outdated+, we do not check the metastore. Instead, > comparison is based on valid write id lists. > - For MVs that +can be outdated+, we still rely on the invalidation cache. > ** The window for valid outdated MVs can be specified in intervals of 1 > minute (less than that, it is difficult to have any guarantees about whether > the MV is actually outdated by less than a minute or not). > ** The async loading is done every interval / 2 (or probably better, we can > make it configurable). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Attachment: HIVE-19024.01.patch > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19024.01.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Status: Patch Available (was: Open) > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19024.01.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Summary: Vectorization: Disable complex type constants for VectorUDFAdaptor (was: Vectorization: Disable complex type constants) > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Status: Open (was: Patch Available) > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19026) Configurable serde for druid kafka indexing
[ https://issues.apache.org/jira/browse/HIVE-19026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reassigned HIVE-19026: --- > Configurable serde for druid kafka indexing > > > Key: HIVE-19026 > URL: https://issues.apache.org/jira/browse/HIVE-19026 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > > https://issues.apache.org/jira/browse/HIVE-18976 introduces support for > setting up druid kafka-indexing service. > Input serialization should be configurable. for now we can say we only > support json, but there should be a mechanism to support other formats. > Perhaps, we can make use of Hive's serde library like LazySimpleSerde etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19018:
Status: Open (was: Patch Available)
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19018:
Status: Patch Available (was: Open)
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18745) Fix MetaStore creation in tests, so multiple MetaStores can be started on the same machine
[
https://issues.apache.org/jira/browse/HIVE-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410196#comment-16410196
]
Sahil Takiar commented on HIVE-18745:
-
+1 LGTM
> Fix MetaStore creation in tests, so multiple MetaStores can be started on the
> same machine
> --
>
> Key: HIVE-18745
> URL: https://issues.apache.org/jira/browse/HIVE-18745
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18745.1.patch, HIVE-18745.2.patch,
> HIVE-18745.3.patch, HIVE-18745.4.patch, HIVE-18745.5.patch,
> HIVE-18745.6.patch, HIVE-18745.7.patch
>
>
> [~janulatha] fixed the problem, when multiple MetaStore tests are started on
> the same machine, then they tried to reserve the same port. This caused
> flakiness in the MetaStore tests run with the ptest framework. See: HIVE-18147
> I reviewed the HIVE-17980, and tried to make sure, that the fix remains in
> every codepath. I was unsuccessful in it. :(
> This Jira aims to go through the MetaStore tests, and make sure all of them
> is using the {{startMetaStoreWithRetry}} method so the different tests will
> not cause each other to fail. Also there were clashes not only in port
> numbers, but warehouse directories as well, so this Jira should fix that also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19021) WM counters are not properly propagated from LLAP to AM
[ https://issues.apache.org/jira/browse/HIVE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410192#comment-16410192 ] Sergey Shelukhin commented on HIVE-19021: - Tested on a cluster, counters are now propagated correctly. > WM counters are not properly propagated from LLAP to AM > --- > > Key: HIVE-19021 > URL: https://issues.apache.org/jira/browse/HIVE-19021 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19021.01.patch, HIVE-19021.02.patch, > HIVE-19021.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19025) spurious ACID logs from HS2
[
https://issues.apache.org/jira/browse/HIVE-19025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-19025:
---
> spurious ACID logs from HS2
> ---
>
> Key: HIVE-19025
> URL: https://issues.apache.org/jira/browse/HIVE-19025
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Major
>
> I'm running some version close to current master, and see the following in
> HS2 stdout.
> I'm running a simple select query with no errors and no special transactional
> logic. Nothing else is running.
> {noformat}
> 18/03/22 15:55:34 INFO client.RMProxy: Connecting to ResourceManager at [snip]
> OK
> Error rolling back: Can't call rollback when autocommit=true
> ...
> 18/03/22 15:56:26 INFO reducesink.VectorReduceSinkObjectHashOperator:
> VectorReduceSinkObjectHashOperator constructor vectorReduceSinkInfo
> org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@4124cdaa
> Error rolling back: Can't call rollback when autocommit=true
> Query ID = sershe_20180322155619_4c58bfa4-ff93-4d4f-8a11-7ddd65c5d2c6
> Total jobs = 1
> Launching Job 1 out of 1
> Error rolling back: Can't call rollback when autocommit=true
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18745) Fix MetaStore creation in tests, so multiple MetaStores can be started on the same machine
[
https://issues.apache.org/jira/browse/HIVE-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410187#comment-16410187
]
Hive QA commented on HIVE-18745:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12915687/HIVE-18745.7.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 334 failed/errored test(s), 13484 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Assigned] (HIVE-19024) Vectorization: Disable complex type constants
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-19024: --- > Vectorization: Disable complex type constants > - > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone
[
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410156#comment-16410156
]
Piotr Findeisen commented on HIVE-18925:
if this all is ok, how do i proceed to have the patch merged?
> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
> Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Assignee: Piotr Findeisen
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-18925.patch
>
>
> Hive Server2 doesn't work if started with
> {{-Duser.timezone=America/Bahia_Banderas}}
>
> Steps to reproduce
> # use [https://github.com/big-data-europe/docker-hive]
> # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to
> {{hive-server}} docker container environment configuration
> # {{docker-compose up}}
> #
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with
> org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with
> org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think
> it's initialization of
> {{org.apache.hive.common.util.TimestampParser#startingDateValue}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19021) WM counters are not properly propagated from LLAP to AM
[ https://issues.apache.org/jira/browse/HIVE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19021: Attachment: HIVE-19021.02.patch > WM counters are not properly propagated from LLAP to AM > --- > > Key: HIVE-19021 > URL: https://issues.apache.org/jira/browse/HIVE-19021 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19021.01.patch, HIVE-19021.02.patch, > HIVE-19021.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19021) WM counters are not properly propagated from LLAP to AM
[ https://issues.apache.org/jira/browse/HIVE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410141#comment-16410141 ] Prasanth Jayachandran commented on HIVE-19021: -- nit comments. looks good to me, +1 > WM counters are not properly propagated from LLAP to AM > --- > > Key: HIVE-19021 > URL: https://issues.apache.org/jira/browse/HIVE-19021 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19021.01.patch, HIVE-19021.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18745) Fix MetaStore creation in tests, so multiple MetaStores can be started on the same machine
[
https://issues.apache.org/jira/browse/HIVE-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410135#comment-16410135
]
Hive QA commented on HIVE-18745:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch java-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 646
unchanged - 1 fixed = 646 total (was 647) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hive-unit-hadoop2 passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9773/dev-support/hive-personality.sh
|
| git revision | master / 560bca0 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9773/yetus/patch-asflicense-problems.txt
|
| modules | C: hcatalog/core hcatalog/webhcat/java-client itests/hive-unit
itests/hive-unit-hadoop2 itests/util ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9773/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix MetaStore creation in tests, so multiple MetaStores can be started on the
> same machine
> --
>
> Key: HIVE-18745
> URL: https://issues.apache.org/jira/browse/HIVE-18745
> Project: Hive
> Issue Type: Sub-task
>
[jira] [Commented] (HIVE-19021) WM counters are not properly propagated from LLAP to AM
[ https://issues.apache.org/jira/browse/HIVE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410125#comment-16410125 ] Sergey Shelukhin commented on HIVE-19021: - Thanks! I changed the approach to dump to TezCounters when actually sending, and also to finalize the counters before the last send. There are lots of null checks already in the code, I'm just basing off those. I think this may happen in tests, e.g. sometimes it would not create TezCounters and log that there's no fragment ID. Legacy :( > WM counters are not properly propagated from LLAP to AM > --- > > Key: HIVE-19021 > URL: https://issues.apache.org/jira/browse/HIVE-19021 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19021.01.patch, HIVE-19021.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19021) WM counters are not properly propagated from LLAP to AM
[ https://issues.apache.org/jira/browse/HIVE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19021: Attachment: HIVE-19021.01.patch > WM counters are not properly propagated from LLAP to AM > --- > > Key: HIVE-19021 > URL: https://issues.apache.org/jira/browse/HIVE-19021 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19021.01.patch, HIVE-19021.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18991) Drop database cascade doesn't work with materialized views
[
https://issues.apache.org/jira/browse/HIVE-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410055#comment-16410055
]
Jesus Camacho Rodriguez commented on HIVE-18991:
[~alangates], uploaded a new patch that brings the code in
{{HiveMetaStoreClient}} back. If I understand that logic correctly, I think we
should not handle the drop database statement from the client in that fashion.
The reason is that triggering individual drop statements for each table from
the client may lead to dropping only some tables, while some other tables and
the database are not dropped (no rollback). I did not change any logic right
now, but I left a note in the code.
Concerning dropping tables that are referenced by MVs, I realized that it
should not be an issue since we will get the foreign key violation that you
were seeing when you opened this case. As a follow-up work, we should probably
print an understandable message instead of the constraint violation sent by
metastore db.
> Drop database cascade doesn't work with materialized views
> --
>
> Key: HIVE-18991
> URL: https://issues.apache.org/jira/browse/HIVE-18991
> Project: Hive
> Issue Type: Bug
> Components: Materialized views, Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18991.01.patch, HIVE-18991.patch
>
>
> Create a database, add a table and then a materialized view that depends on
> the table. Then drop the database with cascade set. Sometimes this will
> fail because when HiveMetaStore.drop_database_core goes to drop all of the
> tables it may drop the base table before the materialized view, which will
> cause an integrity constraint violation in the RDBMS. To resolve this that
> method should change to fetch and drop materialized views before tables.
> cc [~jcamachorodriguez]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19021) WM counters are not properly propagated from LLAP to AM
[ https://issues.apache.org/jira/browse/HIVE-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410057#comment-16410057 ] Siddharth Seth commented on HIVE-19021: --- Kind of rusty on some of this. Given Tez counters are only created when the task starts running, the approach looks ok to me. Didn't understand the comment about "Need to update earlier for runtimes". Getting counters before the RunningTask is created - looks like this will require bigger changes. There's a bunch of "null" checks all over the patch, which seem very unnecessary. Also checks to see if the TezCounters are already set, with a log message. Are these counters ever expected to be null? > WM counters are not properly propagated from LLAP to AM > --- > > Key: HIVE-19021 > URL: https://issues.apache.org/jira/browse/HIVE-19021 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19021.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410052#comment-16410052
]
Aihua Xu commented on HIVE-19018:
-
[~ychena] Yeah. With the change, we will support multiple statements. This
patch is to fix a regression to support "select 3" with ';' at the end.
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18991) Drop database cascade doesn't work with materialized views
[ https://issues.apache.org/jira/browse/HIVE-18991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18991: --- Attachment: HIVE-18991.01.patch > Drop database cascade doesn't work with materialized views > -- > > Key: HIVE-18991 > URL: https://issues.apache.org/jira/browse/HIVE-18991 > Project: Hive > Issue Type: Bug > Components: Materialized views, Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18991.01.patch, HIVE-18991.patch > > > Create a database, add a table and then a materialized view that depends on > the table. Then drop the database with cascade set. Sometimes this will > fail because when HiveMetaStore.drop_database_core goes to drop all of the > tables it may drop the base table before the materialized view, which will > cause an integrity constraint violation in the RDBMS. To resolve this that > method should change to fetch and drop materialized views before tables. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-18928:
Status: Patch Available (was: Open)
Fixing the race condition by using ConcurrentHashMap
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-18928:
Attachment: HIVE-18928.1.patch
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18909) Metrics for results cache
[ https://issues.apache.org/jira/browse/HIVE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18909: -- Labels: Metrics (was: ) > Metrics for results cache > - > > Key: HIVE-18909 > URL: https://issues.apache.org/jira/browse/HIVE-18909 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Labels: Metrics > Attachments: HIVE-18909.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18909) Metrics for results cache
[ https://issues.apache.org/jira/browse/HIVE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410022#comment-16410022 ] Jason Dere commented on HIVE-18909: --- RB at https://reviews.apache.org/r/66222/ > Metrics for results cache > - > > Key: HIVE-18909 > URL: https://issues.apache.org/jira/browse/HIVE-18909 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Labels: Metrics > Attachments: HIVE-18909.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18909) Metrics for results cache
[ https://issues.apache.org/jira/browse/HIVE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18909: -- Attachment: HIVE-18909.1.patch > Metrics for results cache > - > > Key: HIVE-18909 > URL: https://issues.apache.org/jira/browse/HIVE-18909 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18909.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18909) Metrics for results cache
[ https://issues.apache.org/jira/browse/HIVE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18909: -- Status: Patch Available (was: Open) > Metrics for results cache > - > > Key: HIVE-18909 > URL: https://issues.apache.org/jira/browse/HIVE-18909 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18909.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18909) Metrics for results cache
[ https://issues.apache.org/jira/browse/HIVE-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere reassigned HIVE-18909: - Assignee: Jason Dere > Metrics for results cache > - > > Key: HIVE-18909 > URL: https://issues.apache.org/jira/browse/HIVE-18909 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18774) ACID: Use the _copy_N files copyNumber as the implicit statement-id
[ https://issues.apache.org/jira/browse/HIVE-18774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18774: - Assignee: Eugene Koifman > ACID: Use the _copy_N files copyNumber as the implicit statement-id > --- > > Key: HIVE-18774 > URL: https://issues.apache.org/jira/browse/HIVE-18774 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Gopal V >Assignee: Eugene Koifman >Priority: Major > > When upgrading flat ORC files to ACID, use the _copy_N numbering as a > statement-id to avoid having to align the row numbering between _copy_1 and > _copy_2 files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409997#comment-16409997
]
Daniel Dai commented on HIVE-18778:
---
Yes, still in background. There are lots of tests have statistics difference in
addition to PREHOOK/POSTHOOK changes. Need to get sometime to clear these out.
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18778.3.patch, HIVE-18778_TestCliDriver.patch,
> HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18755) Modifications to the metastore for catalogs
[ https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-18755: -- Attachment: HIVE-18755.2.patch > Modifications to the metastore for catalogs > --- > > Key: HIVE-18755 > URL: https://issues.apache.org/jira/browse/HIVE-18755 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch > > > Step 1 of adding catalogs is to add support in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18967) Standalone metastore SQL upgrade scripts do not properly set schema version
[ https://issues.apache.org/jira/browse/HIVE-18967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409970#comment-16409970 ] Thejas M Nair commented on HIVE-18967: -- +1 > Standalone metastore SQL upgrade scripts do not properly set schema version > --- > > Key: HIVE-18967 > URL: https://issues.apache.org/jira/browse/HIVE-18967 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18967.patch > > > The new combined upgrade scripts for Hive 2.3 to 3.0 transition do not > properly set the schema version after they have completed the upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
