[jira] [Updated] (HIVE-18953) Implement CHECK constraint
[ https://issues.apache.org/jira/browse/HIVE-18953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18953: --- Status: Patch Available (was: Open) > Implement CHECK constraint > -- > > Key: HIVE-18953 > URL: https://issues.apache.org/jira/browse/HIVE-18953 > Project: Hive > Issue Type: New Feature >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18953.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18953) Implement CHECK constraint
[ https://issues.apache.org/jira/browse/HIVE-18953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18953: --- Attachment: HIVE-18953.1.patch > Implement CHECK constraint > -- > > Key: HIVE-18953 > URL: https://issues.apache.org/jira/browse/HIVE-18953 > Project: Hive > Issue Type: New Feature >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18953.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18953) Implement CHECK constraint
[ https://issues.apache.org/jira/browse/HIVE-18953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-18953: -- > Implement CHECK constraint > -- > > Key: HIVE-18953 > URL: https://issues.apache.org/jira/browse/HIVE-18953 > Project: Hive > Issue Type: New Feature >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18867) create_with_constraints_duplicate_name and default_constraint_invalid_default_value_length failing
[
https://issues.apache.org/jira/browse/HIVE-18867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18867:
---
Fix Version/s: 3.0.0
> create_with_constraints_duplicate_name and
> default_constraint_invalid_default_value_length failing
> ---
>
> Key: HIVE-18867
> URL: https://issues.apache.org/jira/browse/HIVE-18867
> Project: Hive
> Issue Type: Test
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18867.1.patch
>
>
> The output file for both of these need to be updated
> {noformat}
> Client Execution succeeded but contained differences (error code = 1) after
> executing default_constraint_invalid_default_value_length.q
> 1c1
> < FAILED: SemanticException [Error 10326]: Invalid Constraint syntax Invalid
> Default value:
> '12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234'
> .Maximum character length allowed is 255 .
> ---
> > FAILED: SemanticException [Error 10326]: Invalid Constraint syntax Invalid
> > Default value:
> > '12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234'Maximum
> > character length allowed is 255 .
> {noformat}
> {noformat}
> Client Execution succeeded but contained differences (error code = 1) after
> executing create_with_constraints_duplicate_name.q
> 13c13
> < FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> InvalidObjectException(message:Constraint name already exists: pk1)
> ---
> > FAILED: Execution Error, return code 1 from
> > org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:One or more
> > instances could not be made persistent)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18940) Hive notifications serialize all write DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398036#comment-16398036
]
Alexander Kolbasov commented on HIVE-18940:
---
[~thejas] Yes, you are correct, this is a nice assumption to work from, the
only problem is that there is no way we can guarantee it *and* avoid global
synchronization of all write HMS events. This means that we need to think about
alternative solutions that can relax the accepted behavior of notification
stream. We might provide both 'strict' and 'relaxed' modes, which is Ok, but
complicates HMS even more because we may need different tables and HM methods
to implement the two.
> Hive notifications serialize all write DDL operations
> -
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397992#comment-16397992
]
Hive QA commented on HIVE-18825:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914340/HIVE-18825.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13015 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Updated] (HIVE-18951) Fix the llapdump usage error in llapdump.sh
[ https://issues.apache.org/jira/browse/HIVE-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18951: Status: Patch Available (was: Open) > Fix the llapdump usage error in llapdump.sh > --- > > Key: HIVE-18951 > URL: https://issues.apache.org/jira/browse/HIVE-18951 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Priority: Minor > Attachments: HIVE-18951.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18951) Fix the llapdump usage error in llapdump.sh
[ https://issues.apache.org/jira/browse/HIVE-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang reassigned HIVE-18951: --- Assignee: Saijin Huang > Fix the llapdump usage error in llapdump.sh > --- > > Key: HIVE-18951 > URL: https://issues.apache.org/jira/browse/HIVE-18951 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Attachments: HIVE-18951.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18951) Fix the llapdump usage error in llapdump.sh
[ https://issues.apache.org/jira/browse/HIVE-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18951: Attachment: (was: LICENSE) > Fix the llapdump usage error in llapdump.sh > --- > > Key: HIVE-18951 > URL: https://issues.apache.org/jira/browse/HIVE-18951 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Priority: Minor > Attachments: HIVE-18951.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18951) Fix the llapdump usage error in llapdump.sh
[ https://issues.apache.org/jira/browse/HIVE-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18951: Attachment: HIVE-18951.1.patch > Fix the llapdump usage error in llapdump.sh > --- > > Key: HIVE-18951 > URL: https://issues.apache.org/jira/browse/HIVE-18951 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Priority: Minor > Attachments: HIVE-18951.1.patch, LICENSE > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18951) Fix the llapdump usage error in llapdump.sh
[ https://issues.apache.org/jira/browse/HIVE-18951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18951: Attachment: LICENSE > Fix the llapdump usage error in llapdump.sh > --- > > Key: HIVE-18951 > URL: https://issues.apache.org/jira/browse/HIVE-18951 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Priority: Minor > Attachments: HIVE-18951.1.patch, LICENSE > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397951#comment-16397951
]
Hive QA commented on HIVE-18825:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 20 new + 558 unchanged - 17
fixed = 578 total (was 575) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9626/dev-support/hive-personality.sh
|
| git revision | master / d977a9d |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9626/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9626/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9626/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18915) Better client logging when a HoS session can't be opened
[
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18915:
Status: Patch Available (was: Open)
> Better client logging when a HoS session can't be opened
>
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client
> for Spark session [id]}} when a HoS session can't be opened, would be better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18915) Better client logging when a HoS session can't be opened
[
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397940#comment-16397940
]
Aihua Xu commented on HIVE-18915:
-
We are throwing the proper exception but the Driver just doesn't display it. We
need to show the cause for the exception. I will do more tests.
> Better client logging when a HoS session can't be opened
>
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client
> for Spark session [id]}} when a HoS session can't be opened, would be better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18915) Better client logging when a HoS session can't be opened
[
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18915:
Attachment: HIVE-18915.1.patch
> Better client logging when a HoS session can't be opened
>
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-18915.1.patch
>
>
> Users just get a {{FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client
> for Spark session [id]}} when a HoS session can't be opened, would be better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18763) VectorMapOperator should take into account partition->table serde conversion for all cases
[
https://issues.apache.org/jira/browse/HIVE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397927#comment-16397927
]
Sergey Shelukhin commented on HIVE-18763:
-
Ok I was finally able to return to this.
Looks like it's a much bigger change than I expected.
I included the test that returns different results for different cases and in
fact fails in LLAP IO with either ORC or text...
I cannot repro the issue with Parquet anymore, at least not for these
conversions.
For now I won't be working on this anymore... I think ConvertTreeReader-s
conversion aspects need to be moved from ORC to Hive, since they should not be
ORC specific. Or at least conversion logic should be refactored to be reusable
in wider range of cases. cc [~mmccline]
> VectorMapOperator should take into account partition->table serde conversion
> for all cases
> --
>
> Key: HIVE-18763
> URL: https://issues.apache.org/jira/browse/HIVE-18763
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18763.WIP.patch
>
>
> When table and partition schema differ, non-vectorized MapOperator does row
> by row conversion from whatever is read to the table schema.
> VectorMapOperator is less consistent... it does the conversion as part of
> populating VRBs in row/serde modes (used to read e.g. text row-by-row or
> natively, and make VRBs); see VectorDeserializeRow class convert... methods
> for an example. However, the native VRB mode relies on ORC
> ConvertTreeReader... stuff that lives in ORC, and so never converts anything
> nside VMO.
> So, anything running in native VRB mode that is not the vanilla ORC reader
> will produce data with incorrect schema if there were schema changes and
> partitions are present - there are two such cases right now, LLAP IO with
> ORC or text data, and Parquet.
> It's possible to extend ConvertTreeReader... stuff to LLAP IO ORC that
> already uses TreeReader-s for everything; LLAP IO text and (non-LLAP)
> Parquet, as well as any future users however will have to invent their own
> conversion.
> Therefore, I think the best fix for this is to treat all inputs in VMO the
> same and convert them by default, like the regular MapOperator; and make ORC
> special mode an exception that allows it to bypass the conversion.
> cc [~mmccline]
> Test case - varchar column length should be limited after alter table but it
> isn't.
> {noformat}
> CREATE TABLE schema_evolution_data(insert_num int, boolean1 boolean, tinyint1
> tinyint, smallint1 smallint, int1 int, bigint1 bigint, decimal1
> decimal(38,18), float1 float, double1 double, string1 varchar(50), string2
> varchar(50), date1 date, timestamp1 timestamp, boolean_str string,
> tinyint_str string, smallint_str string, int_str string, bigint_str string,
> decimal_str string, float_str string, double_str string, date_str string,
> timestamp_str string, filler string)
> row format delimited fields terminated by '|' stored as textfile;
> load data local inpath
> '../../data/files/schema_evolution/schema_evolution_data.txt' overwrite into
> table schema_evolution_data;
> drop table if exists vsp;
> create table vsp(vs varchar(50)) partitioned by(s varchar(50)) stored as
> textfile;
> insert into table vsp partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp change column vs vs varchar(3);
> drop table if exists vsp_orc;
> create table vsp_orc(vs varchar(50)) partitioned by(s varchar(50)) stored as
> orc;
> insert into table vsp_orc partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_orc change column vs vs varchar(3);
> drop table if exists vsp_parquet;
> create table vsp_parquet(vs varchar(50)) partitioned by(s varchar(50)) stored
> as parquet;
> insert into table vsp_parquet partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_parquet change column vs vs varchar(3);
> SET hive.llap.io.enabled=true;
> -- BAD results from all queries; parquet affected regardless of IO.
> select length(vs) from vsp;
> select length(vs) from vsp_orc;
> select length(vs) from vsp_parquet;
> SET hive.llap.io.enabled=false;
> select length(vs) from vsp; -- ok
> select length(vs) from vsp_orc; -- ok
> select length(vs) from vsp_parquet; -- still bad
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18763) VectorMapOperator should take into account partition->table serde conversion for all cases
[
https://issues.apache.org/jira/browse/HIVE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-18763:
---
Assignee: (was: Sergey Shelukhin)
> VectorMapOperator should take into account partition->table serde conversion
> for all cases
> --
>
> Key: HIVE-18763
> URL: https://issues.apache.org/jira/browse/HIVE-18763
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18763.WIP.patch
>
>
> When table and partition schema differ, non-vectorized MapOperator does row
> by row conversion from whatever is read to the table schema.
> VectorMapOperator is less consistent... it does the conversion as part of
> populating VRBs in row/serde modes (used to read e.g. text row-by-row or
> natively, and make VRBs); see VectorDeserializeRow class convert... methods
> for an example. However, the native VRB mode relies on ORC
> ConvertTreeReader... stuff that lives in ORC, and so never converts anything
> nside VMO.
> So, anything running in native VRB mode that is not the vanilla ORC reader
> will produce data with incorrect schema if there were schema changes and
> partitions are present - there are two such cases right now, LLAP IO with
> ORC or text data, and Parquet.
> It's possible to extend ConvertTreeReader... stuff to LLAP IO ORC that
> already uses TreeReader-s for everything; LLAP IO text and (non-LLAP)
> Parquet, as well as any future users however will have to invent their own
> conversion.
> Therefore, I think the best fix for this is to treat all inputs in VMO the
> same and convert them by default, like the regular MapOperator; and make ORC
> special mode an exception that allows it to bypass the conversion.
> cc [~mmccline]
> Test case - varchar column length should be limited after alter table but it
> isn't.
> {noformat}
> CREATE TABLE schema_evolution_data(insert_num int, boolean1 boolean, tinyint1
> tinyint, smallint1 smallint, int1 int, bigint1 bigint, decimal1
> decimal(38,18), float1 float, double1 double, string1 varchar(50), string2
> varchar(50), date1 date, timestamp1 timestamp, boolean_str string,
> tinyint_str string, smallint_str string, int_str string, bigint_str string,
> decimal_str string, float_str string, double_str string, date_str string,
> timestamp_str string, filler string)
> row format delimited fields terminated by '|' stored as textfile;
> load data local inpath
> '../../data/files/schema_evolution/schema_evolution_data.txt' overwrite into
> table schema_evolution_data;
> drop table if exists vsp;
> create table vsp(vs varchar(50)) partitioned by(s varchar(50)) stored as
> textfile;
> insert into table vsp partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp change column vs vs varchar(3);
> drop table if exists vsp_orc;
> create table vsp_orc(vs varchar(50)) partitioned by(s varchar(50)) stored as
> orc;
> insert into table vsp_orc partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_orc change column vs vs varchar(3);
> drop table if exists vsp_parquet;
> create table vsp_parquet(vs varchar(50)) partitioned by(s varchar(50)) stored
> as parquet;
> insert into table vsp_parquet partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_parquet change column vs vs varchar(3);
> SET hive.llap.io.enabled=true;
> -- BAD results from all queries; parquet affected regardless of IO.
> select length(vs) from vsp;
> select length(vs) from vsp_orc;
> select length(vs) from vsp_parquet;
> SET hive.llap.io.enabled=false;
> select length(vs) from vsp; -- ok
> select length(vs) from vsp_orc; -- ok
> select length(vs) from vsp_parquet; -- still bad
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18763) VectorMapOperator should take into account partition->table serde conversion for all cases
[
https://issues.apache.org/jira/browse/HIVE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18763:
Attachment: HIVE-18763.WIP.patch
> VectorMapOperator should take into account partition->table serde conversion
> for all cases
> --
>
> Key: HIVE-18763
> URL: https://issues.apache.org/jira/browse/HIVE-18763
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18763.WIP.patch
>
>
> When table and partition schema differ, non-vectorized MapOperator does row
> by row conversion from whatever is read to the table schema.
> VectorMapOperator is less consistent... it does the conversion as part of
> populating VRBs in row/serde modes (used to read e.g. text row-by-row or
> natively, and make VRBs); see VectorDeserializeRow class convert... methods
> for an example. However, the native VRB mode relies on ORC
> ConvertTreeReader... stuff that lives in ORC, and so never converts anything
> nside VMO.
> So, anything running in native VRB mode that is not the vanilla ORC reader
> will produce data with incorrect schema if there were schema changes and
> partitions are present - there are two such cases right now, LLAP IO with
> ORC or text data, and Parquet.
> It's possible to extend ConvertTreeReader... stuff to LLAP IO ORC that
> already uses TreeReader-s for everything; LLAP IO text and (non-LLAP)
> Parquet, as well as any future users however will have to invent their own
> conversion.
> Therefore, I think the best fix for this is to treat all inputs in VMO the
> same and convert them by default, like the regular MapOperator; and make ORC
> special mode an exception that allows it to bypass the conversion.
> cc [~mmccline]
> Test case - varchar column length should be limited after alter table but it
> isn't.
> {noformat}
> CREATE TABLE schema_evolution_data(insert_num int, boolean1 boolean, tinyint1
> tinyint, smallint1 smallint, int1 int, bigint1 bigint, decimal1
> decimal(38,18), float1 float, double1 double, string1 varchar(50), string2
> varchar(50), date1 date, timestamp1 timestamp, boolean_str string,
> tinyint_str string, smallint_str string, int_str string, bigint_str string,
> decimal_str string, float_str string, double_str string, date_str string,
> timestamp_str string, filler string)
> row format delimited fields terminated by '|' stored as textfile;
> load data local inpath
> '../../data/files/schema_evolution/schema_evolution_data.txt' overwrite into
> table schema_evolution_data;
> drop table if exists vsp;
> create table vsp(vs varchar(50)) partitioned by(s varchar(50)) stored as
> textfile;
> insert into table vsp partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp change column vs vs varchar(3);
> drop table if exists vsp_orc;
> create table vsp_orc(vs varchar(50)) partitioned by(s varchar(50)) stored as
> orc;
> insert into table vsp_orc partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_orc change column vs vs varchar(3);
> drop table if exists vsp_parquet;
> create table vsp_parquet(vs varchar(50)) partitioned by(s varchar(50)) stored
> as parquet;
> insert into table vsp_parquet partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_parquet change column vs vs varchar(3);
> SET hive.llap.io.enabled=true;
> -- BAD results from all queries; parquet affected regardless of IO.
> select length(vs) from vsp;
> select length(vs) from vsp_orc;
> select length(vs) from vsp_parquet;
> SET hive.llap.io.enabled=false;
> select length(vs) from vsp; -- ok
> select length(vs) from vsp_orc; -- ok
> select length(vs) from vsp_parquet; -- still bad
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18763) VectorMapOperator should take into account partition->table serde conversion for all cases
[
https://issues.apache.org/jira/browse/HIVE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18763:
Attachment: (was: HIVE-18763.WIP.patch)
> VectorMapOperator should take into account partition->table serde conversion
> for all cases
> --
>
> Key: HIVE-18763
> URL: https://issues.apache.org/jira/browse/HIVE-18763
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> When table and partition schema differ, non-vectorized MapOperator does row
> by row conversion from whatever is read to the table schema.
> VectorMapOperator is less consistent... it does the conversion as part of
> populating VRBs in row/serde modes (used to read e.g. text row-by-row or
> natively, and make VRBs); see VectorDeserializeRow class convert... methods
> for an example. However, the native VRB mode relies on ORC
> ConvertTreeReader... stuff that lives in ORC, and so never converts anything
> nside VMO.
> So, anything running in native VRB mode that is not the vanilla ORC reader
> will produce data with incorrect schema if there were schema changes and
> partitions are present - there are two such cases right now, LLAP IO with
> ORC or text data, and Parquet.
> It's possible to extend ConvertTreeReader... stuff to LLAP IO ORC that
> already uses TreeReader-s for everything; LLAP IO text and (non-LLAP)
> Parquet, as well as any future users however will have to invent their own
> conversion.
> Therefore, I think the best fix for this is to treat all inputs in VMO the
> same and convert them by default, like the regular MapOperator; and make ORC
> special mode an exception that allows it to bypass the conversion.
> cc [~mmccline]
> Test case - varchar column length should be limited after alter table but it
> isn't.
> {noformat}
> CREATE TABLE schema_evolution_data(insert_num int, boolean1 boolean, tinyint1
> tinyint, smallint1 smallint, int1 int, bigint1 bigint, decimal1
> decimal(38,18), float1 float, double1 double, string1 varchar(50), string2
> varchar(50), date1 date, timestamp1 timestamp, boolean_str string,
> tinyint_str string, smallint_str string, int_str string, bigint_str string,
> decimal_str string, float_str string, double_str string, date_str string,
> timestamp_str string, filler string)
> row format delimited fields terminated by '|' stored as textfile;
> load data local inpath
> '../../data/files/schema_evolution/schema_evolution_data.txt' overwrite into
> table schema_evolution_data;
> drop table if exists vsp;
> create table vsp(vs varchar(50)) partitioned by(s varchar(50)) stored as
> textfile;
> insert into table vsp partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp change column vs vs varchar(3);
> drop table if exists vsp_orc;
> create table vsp_orc(vs varchar(50)) partitioned by(s varchar(50)) stored as
> orc;
> insert into table vsp_orc partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_orc change column vs vs varchar(3);
> drop table if exists vsp_parquet;
> create table vsp_parquet(vs varchar(50)) partitioned by(s varchar(50)) stored
> as parquet;
> insert into table vsp_parquet partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_parquet change column vs vs varchar(3);
> SET hive.llap.io.enabled=true;
> -- BAD results from all queries; parquet affected regardless of IO.
> select length(vs) from vsp;
> select length(vs) from vsp_orc;
> select length(vs) from vsp_parquet;
> SET hive.llap.io.enabled=false;
> select length(vs) from vsp; -- ok
> select length(vs) from vsp_orc; -- ok
> select length(vs) from vsp_parquet; -- still bad
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18763) VectorMapOperator should take into account partition->table serde conversion for all cases
[
https://issues.apache.org/jira/browse/HIVE-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18763:
Attachment: HIVE-18763.WIP.patch
> VectorMapOperator should take into account partition->table serde conversion
> for all cases
> --
>
> Key: HIVE-18763
> URL: https://issues.apache.org/jira/browse/HIVE-18763
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18763.WIP.patch
>
>
> When table and partition schema differ, non-vectorized MapOperator does row
> by row conversion from whatever is read to the table schema.
> VectorMapOperator is less consistent... it does the conversion as part of
> populating VRBs in row/serde modes (used to read e.g. text row-by-row or
> natively, and make VRBs); see VectorDeserializeRow class convert... methods
> for an example. However, the native VRB mode relies on ORC
> ConvertTreeReader... stuff that lives in ORC, and so never converts anything
> nside VMO.
> So, anything running in native VRB mode that is not the vanilla ORC reader
> will produce data with incorrect schema if there were schema changes and
> partitions are present - there are two such cases right now, LLAP IO with
> ORC or text data, and Parquet.
> It's possible to extend ConvertTreeReader... stuff to LLAP IO ORC that
> already uses TreeReader-s for everything; LLAP IO text and (non-LLAP)
> Parquet, as well as any future users however will have to invent their own
> conversion.
> Therefore, I think the best fix for this is to treat all inputs in VMO the
> same and convert them by default, like the regular MapOperator; and make ORC
> special mode an exception that allows it to bypass the conversion.
> cc [~mmccline]
> Test case - varchar column length should be limited after alter table but it
> isn't.
> {noformat}
> CREATE TABLE schema_evolution_data(insert_num int, boolean1 boolean, tinyint1
> tinyint, smallint1 smallint, int1 int, bigint1 bigint, decimal1
> decimal(38,18), float1 float, double1 double, string1 varchar(50), string2
> varchar(50), date1 date, timestamp1 timestamp, boolean_str string,
> tinyint_str string, smallint_str string, int_str string, bigint_str string,
> decimal_str string, float_str string, double_str string, date_str string,
> timestamp_str string, filler string)
> row format delimited fields terminated by '|' stored as textfile;
> load data local inpath
> '../../data/files/schema_evolution/schema_evolution_data.txt' overwrite into
> table schema_evolution_data;
> drop table if exists vsp;
> create table vsp(vs varchar(50)) partitioned by(s varchar(50)) stored as
> textfile;
> insert into table vsp partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp change column vs vs varchar(3);
> drop table if exists vsp_orc;
> create table vsp_orc(vs varchar(50)) partitioned by(s varchar(50)) stored as
> orc;
> insert into table vsp_orc partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_orc change column vs vs varchar(3);
> drop table if exists vsp_parquet;
> create table vsp_parquet(vs varchar(50)) partitioned by(s varchar(50)) stored
> as parquet;
> insert into table vsp_parquet partition(s='positive') select string1 from
> schema_evolution_data;
> alter table vsp_parquet change column vs vs varchar(3);
> SET hive.llap.io.enabled=true;
> -- BAD results from all queries; parquet affected regardless of IO.
> select length(vs) from vsp;
> select length(vs) from vsp_orc;
> select length(vs) from vsp_parquet;
> SET hive.llap.io.enabled=false;
> select length(vs) from vsp; -- ok
> select length(vs) from vsp_orc; -- ok
> select length(vs) from vsp_parquet; -- still bad
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: In Progress) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, > HIVE-18908.06.patch, JOIN to MAPJOIN Transformation.pdf, SHARED-MEMORY FULL > OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.06.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, > HIVE-18908.06.patch, JOIN to MAPJOIN Transformation.pdf, SHARED-MEMORY FULL > OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.06.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: (was: HIVE-18908.06.patch) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: In Progress (was: Patch Available) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397899#comment-16397899
]
Sahil Takiar commented on HIVE-18343:
-
+1
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch, HIVE-18343.4.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18533: Attachment: HIVE-18533.3.patch > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch, > HIVE-18533.3.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: JOIN to MAPJOIN Transformation.pdf > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, JOIN to > MAPJOIN Transformation.pdf, SHARED-MEMORY FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: SHARED-MEMORY FULL OUTER MapJoin.pdf > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch, SHARED-MEMORY > FULL OUTER MapJoin.pdf > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18915) Better client logging when a HoS session can't be opened
[
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reassigned HIVE-18915:
---
Assignee: Aihua Xu
> Better client logging when a HoS session can't be opened
>
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Aihua Xu
>Priority: Major
>
> Users just get a {{FAILED: Execution Error, return code 30041 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client
> for Spark session [id]}} when a HoS session can't be opened, would be better
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397813#comment-16397813
]
Hive QA commented on HIVE-18264:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914341/HIVE-18264.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13006 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)
[
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397782#comment-16397782
]
kalyan kumar kalvagadda commented on HIVE-18942:
oh, i see. The title of the jira was misleading.
> ALTER TABLE may generate huge event (with all partitions)
> -
>
> Key: HIVE-18942
> URL: https://issues.apache.org/jira/browse/HIVE-18942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
> parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
> EventMessage.EventType.ADD_PARTITION,
> new AddPartitionEvent(newt, parts, true, handler),
> environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event
> will contain all of them. Partition object itself isn't very small either, so
> we may end up with huge events which would be stored and then transmitted
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)
[
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1639#comment-1639
]
Na Li commented on HIVE-18942:
--
[~kkalyan] The event type is "EventMessage.EventType.ADD_PARTITION", not
"EventType.ALTER_TABLE". It puts all partitions in the event.
{code:java}
In DbNotificationListener
public void onAddPartition (AddPartitionEvent partitionEvent)
throws MetaException {
Table t = partitionEvent.getTable();
NotificationEvent event = new NotificationEvent(0, now(),
HCatConstants.HCAT_ADD_PARTITION_EVENT,
msgFactory.buildAddPartitionMessage(t,
partitionEvent.getPartitions()).toString());
event.setDbName(t.getDbName());
event.setTableName(t.getTableName());
enqueue(event, partitionEvent);
}
{code}
> ALTER TABLE may generate huge event (with all partitions)
> -
>
> Key: HIVE-18942
> URL: https://issues.apache.org/jira/browse/HIVE-18942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
> parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
> EventMessage.EventType.ADD_PARTITION,
> new AddPartitionEvent(newt, parts, true, handler),
> environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event
> will contain all of them. Partition object itself isn't very small either, so
> we may end up with huge events which would be stored and then transmitted
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397774#comment-16397774 ] Prasanth Jayachandran commented on HIVE-18933: -- ok. make sense. +1 > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397733#comment-16397733
]
Hive QA commented on HIVE-18264:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} itests/hcatalog-unit: The patch generated 1 new + 22
unchanged - 0 fixed = 23 total (was 22) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
28s{color} | {color:red} standalone-metastore: The patch generated 28 new +
1033 unchanged - 26 fixed = 1061 total (was 1059) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
53s{color} | {color:red} standalone-metastore generated 4 new + 57 unchanged -
2 fixed = 61 total (was 59) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9625/dev-support/hive-personality.sh
|
| git revision | master / d977a9d |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9625/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9625/yetus/diff-checkstyle-itests_hcatalog-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9625/yetus/diff-checkstyle-standalone-metastore.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9625/yetus/diff-javadoc-javadoc-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9625/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hcatalog-unit service standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9625/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CachedStore: Store cached partitions/col stats within the table cache and
> make prewarm non-blocking
> ---
>
> Key: HIVE-18264
> URL: https://issues.apache.org/jira/browse/HIVE-18264
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch,
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch,
>
[jira] [Assigned] (HIVE-18950) DESCRIBE EXTENDED missing details of default constraint
[
https://issues.apache.org/jira/browse/HIVE-18950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aswathy Chellammal Sreekumar reassigned HIVE-18950:
---
> DESCRIBE EXTENDED missing details of default constraint
> ---
>
> Key: HIVE-18950
> URL: https://issues.apache.org/jira/browse/HIVE-18950
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
>
> Describe extended output is missing default constraint details
> {noformat}
> 0: jdbc:hive2://ctr-e138-1518143905142-95188-> create table t1(j int
> constraint c1 default 4);
> INFO : Compiling
> command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318):
> create table t1(j int constraint c1 default 4)
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318);
> Time taken: 0.015 seconds
> INFO : Executing
> command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318):
> create table t1(j int constraint c1 default 4)
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180313202851_de315f0e-4064-467d-9dcc-f8dd7f737318);
> Time taken: 0.048 seconds
> INFO : OK
> No rows affected (0.087 seconds)
> {noformat}
> {noformat}
> 0: jdbc:hive2://ctr-e138-1518143905142-95188-> DESCRIBE EXTENDED t1;
> INFO : Compiling
> command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc):
> DESCRIBE EXTENDED t1
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from
> deserializer), FieldSchema(name:data_type, type:string, comment:from
> deserializer), FieldSchema(name:comment, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc);
> Time taken: 0.029 seconds
> INFO : Executing
> command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc):
> DESCRIBE EXTENDED t1
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180313215805_0596cea8-918c-46f7-bd9a-8611972eb3cc);
> Time taken: 0.03 seconds
> INFO : OK
> +-++--+
> | col_name | data_type
> | comment |
> +-++--+
> | j | int
> | |
> | | NULL
> | NULL |
> | Detailed Table Information | Table(tableName:t1, dbName:default,
> owner:hrt_qa, createTime:1520972931, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:j, type:int, comment:null)],
> location:hdfs://mycluster/apps/hive/warehouse/t1,
> inputFormat:org.apache.hadoop.mapred.TextInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
> partitionKeys:[], parameters:{totalSize=0, numRows=0, rawDataSize=0,
> transactional_properties=insert_only,
> COLUMN_STATS_ACCURATE={\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"j\":\"true\"}},
> numFiles=0, transient_lastDdlTime=1520972931, transactional=true},
> viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
> rewriteEnabled:false) | |
> +-++--+
> 3 rows selected (0.099 seconds){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397671#comment-16397671
]
Hive QA commented on HIVE-18908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914338/HIVE-18908.05.patch
{color:green}SUCCESS:{color} +1 due to 33 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 13038 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397649#comment-16397649 ] Sergey Shelukhin commented on HIVE-18933: - That is one thing that it's doing. It's also removing clone calls. I'm not sure why they are needed... seems error prone to clone an object with IO stream and codec via Java shallow clone without explicitly accounting for things being duplicated. > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397647#comment-16397647 ] Sergey Shelukhin commented on HIVE-18281: - Left comments on RB, mostly minor. I am not sure about timing/races w.r.t. leadership switching. If activities during leadership switch take too long should they time out? etc. I'm hoping/assuming the latch itself has some grace period that should be configurable and/or accounted for. There also appear to be some unrelated logic changes in some places. > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18281.1.patch, HIVE-18281.2.patch, > HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397643#comment-16397643
]
Hive QA commented on HIVE-18908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
45s{color} | {color:red} root: The patch generated 433 new + 4270 unchanged -
155 fixed = 4703 total (was 4425) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} itests/hive-jmh: The patch generated 0 new + 11
unchanged - 6 fixed = 11 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
20s{color} | {color:red} ql: The patch generated 433 new + 3192 unchanged - 149
fixed = 3625 total (was 3341) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9624/dev-support/hive-personality.sh
|
| git revision | master / f9e6522 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9624/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9624/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9624/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api common serde . itests itests/hive-jmh ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9624/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add support for FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Matt
[jira] [Commented] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397603#comment-16397603 ] Prasanth Jayachandran commented on HIVE-18933: -- Is this patch is just removing references to codec object (delegating the cleanup to parent data reader)? > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397598#comment-16397598
]
Sankar Hariappan commented on HIVE-18864:
-
Thanks for the review [~ekoifman]!
Also, the test failures are irrelevant to the patch and it is committed to
master!
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-18864:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397586#comment-16397586
]
Jesus Camacho Rodriguez commented on HIVE-18825:
Uploaded a rebased version of the patch and created HIVE-18948 for the
follow-up work.
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18919) remove separate keytab setting for ZK in LLAP
[ https://issues.apache.org/jira/browse/HIVE-18919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18919: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master after adding to SetProcessor as per an offline comment. Thanks for the review! > remove separate keytab setting for ZK in LLAP > - > > Key: HIVE-18919 > URL: https://issues.apache.org/jira/browse/HIVE-18919 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18919.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Attachment: HIVE-18264.7.patch > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, > HIVE-18264.6.patch, HIVE-18264.7.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397578#comment-16397578 ] Vaibhav Gumashta commented on HIVE-18264: - v7 rebased on master. > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, > HIVE-18264.6.patch, HIVE-18264.7.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18825) Define ValidTxnList before starting query optimization
[
https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18825:
---
Attachment: HIVE-18825.05.patch
> Define ValidTxnList before starting query optimization
> --
>
> Key: HIVE-18825
> URL: https://issues.apache.org/jira/browse/HIVE-18825
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch,
> HIVE-18825.03.patch, HIVE-18825.04.patch, HIVE-18825.05.patch,
> HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened
> after the materialization was created. To compute incremental view
> maintenance, we need to be able to filter only new rows from those base
> tables. That can be done by inserting a filter operator with condition e.g.
> {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId NOT
> IN()}} on top of the MVs query definition and triggering the
> rewriting (which should in turn produce a partial rewriting). However, to do
> that, we need to have a value for {{ValidTxnList}} during query compilation
> so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There
> should not be any visible changes for end user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries/MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Description: Per table write ID implementation (HIVE-18192) maintains a map between txn ID and table write ID in TXN_TO_WRITE_ID meta table. The entries in this table is used to generate ValidWriteIdList for the given ValidTxnList to ensure snapshot isolation. When table or database is dropped, then these entries are cleaned-up. But, it is necessary to clean-up for active tables too for better performance. Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which is referred by any active ValidTxnList snapshot as open/aborted txn. If no references found in this table for any txn, then it is eligible for cleanup. After clean-up, need to maintain just one entry (highest committed txn) per table to mark as LWM (low water mark). was: Per table write ID implementation (HIVE-18192) maintains a map between txn ID and table write ID in TXN_TO_WRITE_ID meta table. The entries in this table is used to generate ValidWriteIdList for the given ValidTxnList to ensure snapshot isolation. When table or database is dropped, then these entries are cleaned-up. But, it is necessary to clean-up for active tables too for better performance. Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which is referred by any active ValidTxnList snapshot as open/aborted txn. If no references found in this table for any txn, then it is eligible for cleanup. After clean-up, need to maintain just one entry per table to mark as LWM (low water mark). > Cleaner for TXN_TO_WRITE_ID table entries/MIN_HISTORY_LEVEL. > > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID > Fix For: 3.0.0 > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > Need to have another table MIN_HISTORY_LEVEL to maintain the least txn which > is referred by any active ValidTxnList snapshot as open/aborted txn. If no > references found in this table for any txn, then it is eligible for cleanup. > After clean-up, need to maintain just one entry (highest committed txn) per > table to mark as LWM (low water mark). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397564#comment-16397564 ] Sergey Shelukhin commented on HIVE-18933: - Negative tests OOMed as usual. [~prasanth_j] can you take a look? > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18947) Add integration tests for HoS + kerberos
[
https://issues.apache.org/jira/browse/HIVE-18947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-18947:
---
> Add integration tests for HoS + kerberos
> -
>
> Key: HIVE-18947
> URL: https://issues.apache.org/jira/browse/HIVE-18947
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> Would be nice to have some unit tests that cover the Kerberos integration in
> {{SparkClientImpl}}. Might be worth exploring adding delegation token, and
> impersonation coverage too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: In Progress) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.05.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch, HIVE-18908.05.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: In Progress (was: Patch Available) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397491#comment-16397491 ] Sahil Takiar commented on HIVE-18922: - CC: [~ashutoshc] > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397472#comment-16397472
]
Eugene Koifman commented on HIVE-18864:
---
+1
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397469#comment-16397469
]
Sankar Hariappan commented on HIVE-18864:
-
String s = "select t2w_txnid, t2w_writeid from TXN_TO_WRITE_ID where t2w_txnid
<= " + txnHwm
+ " and t2w_database = " + quoteString(names[0])
+ " and t2w_table = " + quoteString(names[1])
+ " order by t2w_writeid asc";
In this query, we fetch the list of write ids allocated by txns <= txnHwm... In
my example, writeId=7 is allocated by txnId=txnHwm, but 6 was allocated by
txnId > txnHwm... So, it won't be listed by this query and hence writeId=,
never make it to the open list
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397469#comment-16397469
]
Sankar Hariappan edited comment on HIVE-18864 at 3/13/18 6:56 PM:
--
String s = "select t2w_txnid, t2w_writeid from TXN_TO_WRITE_ID where t2w_txnid
<= " + txnHwm
+ " and t2w_database = " + quoteString(names[0])
+ " and t2w_table = " + quoteString(names[1])
+ " order by t2w_writeid asc";
In this query, we fetch the list of write ids allocated by txns <= txnHwm...
In my example, writeId=7 is allocated by txnId=txnHwm, but 6 was allocated by
txnId > txnHwm... So, it won't be listed by this query and hence writeId=6 will
never make it to the open list
was (Author: sankarh):
String s = "select t2w_txnid, t2w_writeid from TXN_TO_WRITE_ID where t2w_txnid
<= " + txnHwm
+ " and t2w_database = " + quoteString(names[0])
+ " and t2w_table = " + quoteString(names[1])
+ " order by t2w_writeid asc";
In this query, we fetch the list of write ids allocated by txns <= txnHwm... In
my example, writeId=7 is allocated by txnId=txnHwm, but 6 was allocated by
txnId > txnHwm... So, it won't be listed by this query and hence writeId=,
never make it to the open list
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18940) Hive notifications serialize all write DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397463#comment-16397463
]
Thejas M Nair edited comment on HIVE-18940 at 3/13/18 6:54 PM:
---
For replication purposes, and perhaps for sentry delta updates capture as well,
the EVENT_ID has to be in the order of commit.
For example, if the EVENT_ID 5 has been written and then consumed by
replication program, it would then only look for rows where EVENT_ID > 5. So if
there are two concurrent transactions writing new rows and one of them with
EVENT_ID 5 commits before EVENT_ID 4, then EVENT_ID 4 would get missed.
Holes would be OK, what is not OK is that for another application to see row
with EVENT_ID 5 getting visible before one with EVENT_ID 4.
DB generated timestamp has same issue, unless it can represent the commit
sequence.
I believe the use of database autoincrement field was considered in HIVE-16886
and it wasn't meeting this criteria.
cc [~anishek]
was (Author: thejas):
For replication purposes, and perhaps for sentry delta updates capture as well,
the EVENT_ID has to be in the order of commit.
For example, if the EVENT_ID 5 has been written and then consumed by
replication program, it would then only look for rows where EVENT_ID > 5. So if
there are two concurrent transactions writing new rows and one of them with
EVENT_ID 5 commits before EVENT_ID 4, then EVENT_ID 4 would get missed.
Holes would be OK, what is not OK is that for another application to see row
with EVENT_ID 5 getting visible before one with EVENT_ID 4.
I believe the use of database autoincrement field was considered in HIVE-16886
and it wasn't meeting this criteria.
cc [~anishek]
> Hive notifications serialize all write DDL operations
> -
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18940) Hive notifications serialize all write DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397463#comment-16397463
]
Thejas M Nair commented on HIVE-18940:
--
For replication purposes, and perhaps for sentry delta updates capture as well,
the EVENT_ID has to be in the order of commit.
For example, if the EVENT_ID 5 has been written and then consumed by
replication program, it would then only look for rows where EVENT_ID > 5. So if
there are two concurrent transactions writing new rows and one of them with
EVENT_ID 5 commits before EVENT_ID 4, then EVENT_ID 4 would get missed.
Holes would be OK, what is not OK is that for another application to see row
with EVENT_ID 5 getting visible before one with EVENT_ID 4.
I believe the use of database autoincrement field was considered in HIVE-16886
and it wasn't meeting this criteria.
cc [~anishek]
> Hive notifications serialize all write DDL operations
> -
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.
[
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397460#comment-16397460
]
Eugene Koifman commented on HIVE-18864:
---
I'm looking at the original (before this patch) implementation of
getValidWriteIdsForTable() and I'm not sure why it's not sufficient.
txnHwm is fixed for the duration of the txn. Since we recompute
ValidWriteIdList for each statement of a multi-stmt txn, at step 7 in the
Description, you'd see writeId=7 as it belongs to txn10 and writeId=6 would be
considered invalid because it belongs to txn11 which is not isValid() for txn
10.
> ValidWriteIdList snapshot seems incorrect if obtained after allocating
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after
> a read. Let's see the below scenario.
> # Committed txn=9 writes on table T1 with writeId=5.
> # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Open txn=11, writes on table T1 with writeid=6.
> # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
> # Write table T1 from txn=10 with writeId=7.
> # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null,
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of
> ValidWriteIdList based on txn_HWM. Any writeId allocated by txnId > txn_HWM
> should be marked as open. In this example, *ValidWriteIdList(open:6,
> write_HWM=7)* should be generated.{color}
> {color:#33}cc{color} [~ekoifman], [~thejas]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18281) HiveServer2 HA for LLAP and Workload Manager
[ https://issues.apache.org/jira/browse/HIVE-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397459#comment-16397459 ] Prasanth Jayachandran commented on HIVE-18281: -- [~sershe] can you please review this patch? > HiveServer2 HA for LLAP and Workload Manager > > > Key: HIVE-18281 > URL: https://issues.apache.org/jira/browse/HIVE-18281 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18281.1.patch, HIVE-18281.2.patch, > HIVE-18281.WIP.patch, HSI-HA.pdf > > > When running HS2 with LLAP and Workload Manager, HS2 becomes single point of > failure as some of the states for workload management and scheduling are > maintained in-memory. > The proposal is to support Active/Passive mode of high availability in which, > all HS2 and tez AMs registers with ZooKeeper and a leader have to be chosen > which will maintain stateful information. Clients using service discovery > will always connect to the leader for submitting queries. There will also be > some responsibilities for the leader, failover handling, tez session > reconnect etc. Will upload some more detailed information in a separate doc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[
https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397425#comment-16397425
]
Anant Mittal commented on HIVE-18922:
-
It definitely happens with {{MERGE INTO}} queries. I tried some simple non
merge queries and did not see the behavior. You are right, it might be specific
to ACID but I cannot say for sure.
> Hive is not cleaning up staging directories
>
>
> Key: HIVE-18922
> URL: https://issues.apache.org/jira/browse/HIVE-18922
> Project: Hive
> Issue Type: Bug
>Reporter: Anant Mittal
>Priority: Major
>
> Hive is creating hdfs folders with format
> /.hive-staging_hive__-xx/-ext-x
> These are not being cleaned up even after long duration. The folder is used
> to load to the table. Example:
> Loading data to table default.tablename from
> hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1
>
> This might be covered to some extent by HIVE-11940 but, want to make sure all
> cases are addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18668) Really shade guava in ql
[
https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397281#comment-16397281
]
Hive QA commented on HIVE-18668:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914288/HIVE-18668.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 972 failed/errored test(s), 8786 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.org.apache.hadoop.hive.cli.TestBeeLineDriver
(batchId=250)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[buckets]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_database]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_hdfs]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[having]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_move]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_only]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join2]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[load_data]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join_on_filter]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[multiple_agg]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[multiple_db]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[nested_outer_join]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ptf_general_queries]
(batchId=253)
[jira] [Commented] (HIVE-18668) Really shade guava in ql
[
https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397253#comment-16397253
]
Hive QA commented on HIVE-18668:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9623/dev-support/hive-personality.sh
|
| git revision | master / ccc82cf |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9623/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9623/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Really shade guava in ql
>
>
> Key: HIVE-18668
> URL: https://issues.apache.org/jira/browse/HIVE-18668
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18668.01.patch, HIVE-18668.02.patch
>
>
> After HIVE-15393 a test started to fail in druid; after some investigation it
> turned out that ql doesn't shade it's guava artifact at all...because it
> shades 'com.google.guava' instead 'com.google.common'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18888) Replace synchronizedMap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-1: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Alexander for the contribution! > Replace synchronizedMap with ConcurrentHashMap > -- > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0, 2.3.3 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-1.01.patch, HIVE-1.02.patch > > > There are a bunch of places that use Collections.synchronizedMap instead of > ConcurrentHashMap which are better. We should search/replace the uses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17478) Move filesystem stats collection from metastore to ql
[
https://issues.apache.org/jira/browse/HIVE-17478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397218#comment-16397218
]
Hive QA commented on HIVE-17478:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914287/HIVE-17478.07.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 56 failed/errored test(s), 13003 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397187#comment-16397187
]
Ashutosh Chauhan commented on HIVE-18944:
-
+1
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18944.01.patch
>
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.setLongValue(VectorHashKeyWrapperBatch.java:994)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1179)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18919) remove separate keytab setting for ZK in LLAP
[ https://issues.apache.org/jira/browse/HIVE-18919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397184#comment-16397184 ] Ashutosh Chauhan commented on HIVE-18919: - +1 > remove separate keytab setting for ZK in LLAP > - > > Key: HIVE-18919 > URL: https://issues.apache.org/jira/browse/HIVE-18919 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18919.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397151#comment-16397151
]
Zoltan Haindrich commented on HIVE-18944:
-
test failures are not related; will move test from
{{minillaplocal.shared.query.files}} to {{minillaplocal.query.files}} on commit
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18944.01.patch
>
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.setLongValue(VectorHashKeyWrapperBatch.java:994)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1179)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18804) StatsUtils.getColStatisticsFromExprMap may only provide info for a column once
[
https://issues.apache.org/jira/browse/HIVE-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397135#comment-16397135
]
Zoltan Haindrich commented on HIVE-18804:
-
[~ashutoshc] it's the same column; but it turned out that the name is
different; for example at
[constantpropagate|https://github.com/apache/hive/blob/12041d39f052dc8e4858815da15c967cb378fae9/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java#L1223]
the columnExprMap is extended with the '_col' aliased expression - this
eventually starts multiplicating the stats datasize
[here|https://github.com/apache/hive/blob/12041d39f052dc8e4858815da15c967cb378fae9/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1524]
It would be better to fix this the other way around; by removing these
additions of "_col" to the exprmap?
> StatsUtils.getColStatisticsFromExprMap may only provide info for a column once
> --
>
> Key: HIVE-18804
> URL: https://issues.apache.org/jira/browse/HIVE-18804
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18804.01.patch, HIVE-18804.02.patch
>
>
> currently {{StatsUtils.getColStatisticsFromExprMap}} may duplicate the
> datasize by passing the info about the same column more than once
> https://github.com/apache/hive/blob/e8e5ab24616aa834f4966efe3a5f437f6bee4d1d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1529
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17478) Move filesystem stats collection from metastore to ql
[
https://issues.apache.org/jira/browse/HIVE-17478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397118#comment-16397118
]
Hive QA commented on HIVE-17478:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 643 unchanged - 8
fixed = 645 total (was 651) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} standalone-metastore: The patch generated 0 new +
762 unchanged - 20 fixed = 762 total (was 782) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9622/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9622/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9622/yetus/patch-asflicense-problems.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9622/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move filesystem stats collection from metastore to ql
> -
>
> Key: HIVE-17478
> URL: https://issues.apache.org/jira/browse/HIVE-17478
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-17478.01.patch, HIVE-17478.01wip01.patch,
> HIVE-17478.02.patch, HIVE-17478.03.patch, HIVE-17478.04.patch,
> HIVE-17478.05.patch, HIVE-17478.06.patch, HIVE-17478.07.patch
>
>
> filesystem level stats are collected automatically at metastore server
> side...however computing these stats earlier during planning or query
> execution may enable to launch stat collection on a newly added partition
> only if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18668) Really shade guava in ql
[ https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18668: Attachment: HIVE-18668.02.patch > Really shade guava in ql > > > Key: HIVE-18668 > URL: https://issues.apache.org/jira/browse/HIVE-18668 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18668.01.patch, HIVE-18668.02.patch > > > After HIVE-15393 a test started to fail in druid; after some investigation it > turned out that ql doesn't shade it's guava artifact at all...because it > shades 'com.google.guava' instead 'com.google.common' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18668) Really shade guava in ql
[ https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-18668: Status: Patch Available (was: Reopened) > Really shade guava in ql > > > Key: HIVE-18668 > URL: https://issues.apache.org/jira/browse/HIVE-18668 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18668.01.patch, HIVE-18668.02.patch > > > After HIVE-15393 a test started to fail in druid; after some investigation it > turned out that ql doesn't shade it's guava artifact at all...because it > shades 'com.google.guava' instead 'com.google.common' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397083#comment-16397083
]
Hive QA commented on HIVE-18944:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914267/HIVE-18944.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 104 failed/errored test(s), 13788 tests
executed
*Failed tests:*
{noformat}
TestTriggersTezSessionPoolManager - did not produce a TEST-*.xml file (likely
timed out) (batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_groupingset_bug]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_convert]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_truncate_cols]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_select]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_subquery_chain_exists]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[temp_table_rename]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[touch2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure5]
(batchId=95)
[jira] [Updated] (HIVE-17478) Move filesystem stats collection from metastore to ql
[ https://issues.apache.org/jira/browse/HIVE-17478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-17478: Attachment: HIVE-17478.07.patch > Move filesystem stats collection from metastore to ql > - > > Key: HIVE-17478 > URL: https://issues.apache.org/jira/browse/HIVE-17478 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-17478.01.patch, HIVE-17478.01wip01.patch, > HIVE-17478.02.patch, HIVE-17478.03.patch, HIVE-17478.04.patch, > HIVE-17478.05.patch, HIVE-17478.06.patch, HIVE-17478.07.patch > > > filesystem level stats are collected automatically at metastore server > side...however computing these stats earlier during planning or query > execution may enable to launch stat collection on a newly added partition > only if needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18668) Really shade guava in ql
[ https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397053#comment-16397053 ] Zoltan Haindrich commented on HIVE-18668: - yes, my original intention was to relocate the contents; now I see that the shade plugin is currently used to "just" generate a fat tarball...and it only relocates a few pieces; in the current form even an alternate jackson lib could cause problems...but the most fragile from all of these libs is guava... > Really shade guava in ql > > > Key: HIVE-18668 > URL: https://issues.apache.org/jira/browse/HIVE-18668 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18668.01.patch > > > After HIVE-15393 a test started to fail in druid; after some investigation it > turned out that ql doesn't shade it's guava artifact at all...because it > shades 'com.google.guava' instead 'com.google.common' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396991#comment-16396991
]
Hive QA commented on HIVE-18944:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 14 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 30 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9621/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9621/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9621/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9621/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9621/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18944.01.patch
>
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
>
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396956#comment-16396956
]
Hive QA commented on HIVE-18908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914260/HIVE-18908.04.patch
{color:green}SUCCESS:{color} +1 due to 34 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 65 failed/errored test(s), 13038 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396933#comment-16396933
]
Hive QA commented on HIVE-18908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
5s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
27s{color} | {color:red} root: The patch generated 466 new + 4275 unchanged -
150 fixed = 4741 total (was 4425) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} itests/hive-jmh: The patch generated 2 new + 16
unchanged - 1 fixed = 18 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
16s{color} | {color:red} ql: The patch generated 464 new + 3192 unchanged - 149
fixed = 3656 total (was 3341) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9620/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9620/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9620/yetus/diff-checkstyle-itests_hive-jmh.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9620/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9620/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api common serde . itests itests/hive-jmh ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9620/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add support for FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch,
> HIVE-18908.03.patch, HIVE-18908.04.patch
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)
[
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396929#comment-16396929
]
kalyan kumar kalvagadda commented on HIVE-18942:
[~akolb] That may not be the case.
AddPartitionEvent that is generated here is different from the
NotificationEvent that is inserted in the NOTIFICATION_LOG table.
{noformat}
public void onAlterTable(AlterTableEvent tableEvent) throws MetaException {
Table before = tableEvent.getOldTable();
Table after = tableEvent.getNewTable();
NotificationEvent event =
new NotificationEvent(0, now(), EventType.ALTER_TABLE.toString(),
msgFactory
.buildAlterTableMessage(before, after,
tableEvent.getIsTruncateOp()).toString());
event.setDbName(after.getDbName());
event.setTableName(after.getTableName());
process(event, tableEvent);
}
{noformat}
This is how later table JSON is constructed as sentry is interested in below
things in the alter table notification
# Old Db Name
# New Db Name
# Old Table Name
# New Table Name
# Old Location
# New Location
{noformat}
public JSONAlterTableMessage(String server, String servicePrincipal, Table
tableObjBefore, Table tableObjAfter, Long timestamp) {
this.server = server;
this.servicePrincipal = servicePrincipal;
this.db = tableObjBefore.getDbName();
this.table = tableObjBefore.getTableName();
this.timestamp = timestamp;
try {
this.tableObjBeforeJson =
JSONMessageFactory.createTableObjJson(tableObjBefore);
this.tableObjAfterJson =
JSONMessageFactory.createTableObjJson(tableObjAfter);
} catch (TException var7) {
throw new IllegalArgumentException("Could not serialize: ", var7);
}
this.checkValid();
}
{noformat}
I think the NotificationEvent inserted in NOTIFICATION_LOG table may not have
details of the partitions.
> ALTER TABLE may generate huge event (with all partitions)
> -
>
> Key: HIVE-18942
> URL: https://issues.apache.org/jira/browse/HIVE-18942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
> parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
> EventMessage.EventType.ADD_PARTITION,
> new AddPartitionEvent(newt, parts, true, handler),
> environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event
> will contain all of them. Partition object itself isn't very small either, so
> we may end up with huge events which would be stored and then transmitted
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396922#comment-16396922
]
Zoltan Haindrich commented on HIVE-18944:
-
[~ashutoshc] could you please take a look?
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18944.01.patch
>
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.setLongValue(VectorHashKeyWrapperBatch.java:994)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1179)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18944:
Attachment: HIVE-18944.01.patch
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18944.01.patch
>
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.setLongValue(VectorHashKeyWrapperBatch.java:994)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1179)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18944:
Status: Patch Available (was: Open)
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18944.01.patch
>
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.setLongValue(VectorHashKeyWrapperBatch.java:994)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:461)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1179)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[
https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-18944:
Description:
groupingSetsPosition is set to -1 in case there are no grouping sets; however
DPP calls the constructor with 0
this could potentially trigger an unwanted emittion of a summary row
{code}
2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.setLongValue(VectorHashKeyWrapperBatch.java:994)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.close(VectorGroupByOperator.java:461)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1179)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:722)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:746)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
{code}
was:groupingSetsPosition is set to -1 in case there are no grouping sets;
however DPP calls the constructor with 0
> Groupping sets position is set incorrectly during DPP
> -
>
> Key: HIVE-18944
> URL: https://issues.apache.org/jira/browse/HIVE-18944
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> groupingSetsPosition is set to -1 in case there are no grouping sets; however
> DPP calls the constructor with 0
> this could potentially trigger an unwanted emittion of a summary row
> {code}
> 2018-03-13T05:58:16,226 ERROR [TezTR-881987_1_5_1_1_0] tez.TezProcessor:
> java.lang.RuntimeException: Hive Runtime Error while closing operators: 0
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:407)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
>
[jira] [Assigned] (HIVE-18944) Groupping sets position is set incorrectly during DPP
[ https://issues.apache.org/jira/browse/HIVE-18944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-18944: --- > Groupping sets position is set incorrectly during DPP > - > > Key: HIVE-18944 > URL: https://issues.apache.org/jira/browse/HIVE-18944 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > groupingSetsPosition is set to -1 in case there are no grouping sets; however > DPP calls the constructor with 0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18711) Add percentile_cont and percentile_disc udaf
[ https://issues.apache.org/jira/browse/HIVE-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-18711: - Assignee: Miklos Gergely > Add percentile_cont and percentile_disc udaf > > > Key: HIVE-18711 > URL: https://issues.apache.org/jira/browse/HIVE-18711 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Ashutosh Chauhan >Assignee: Miklos Gergely >Priority: Major > > Most common way to implement this is via ordered aggregate which allows users > to specify sort specification with group by clause. Some implementations also > allow to use these with window functions. > Since Hive doesn't have concept of ordered aggregates yet, one possibility is > to support these only for window functions where sort specification is also > taken from window clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.04.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: In Progress) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch, HIVE-18908.04.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: In Progress (was: Patch Available) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396874#comment-16396874
]
Hive QA commented on HIVE-18910:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914229/HIVE-18910.6.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 729 failed/errored test(s), 13806 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=250)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=250)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=250)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=250)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=250)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=250)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=253)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=253)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStats]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_file_format]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table2_h23]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_numbuckets_partitioned_table_h23]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_clusterby_sortby]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_coltype]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_skewed_table]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_add_partition]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_column_stats]
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_not_sorted]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_serde2]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_as_select]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_col_type]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_table_null_partition]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_tbl_date]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_10]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_reordering_values]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_11]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_1]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_3]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_4]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_5]
(batchId=89)
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396843#comment-16396843
]
Hive QA commented on HIVE-18910:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 3 new + 88 unchanged
- 3 fixed = 91 total (was 91) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} serde: The patch generated 142 new + 214 unchanged - 3
fixed = 356 total (was 217) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 33
unchanged - 0 fixed = 34 total (was 33) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 24 new + 1113 unchanged - 3
fixed = 1137 total (was 1116) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 50 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9619/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/patch-mvninstall-hcatalog_streaming.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/diff-checkstyle-hcatalog_streaming.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api serde hcatalog/streaming ql standalone-metastore U:
. |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9619/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396828#comment-16396828
]
Hive QA commented on HIVE-18910:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914229/HIVE-18910.6.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 640 failed/errored test(s), 13013 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[ https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396827#comment-16396827 ] Rui Li commented on HIVE-18034: --- +1 > Improving logging with HoS executors spend lots of time in GC > - > > Key: HIVE-18034 > URL: https://issues.apache.org/jira/browse/HIVE-18034 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18034.1.patch, HIVE-18034.2.patch, > HIVE-18034.3.patch, HIVE-18034.4.patch, HIVE-18034.6.patch, HIVE-18034.7.patch > > > There are times when Spark will spend lots of time doing GC. The Spark > History UI shows a bunch of red flags when too much time is spent in GC. It > would be nice if those warnings are propagated to Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396777#comment-16396777
]
Hive QA commented on HIVE-18910:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
40s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 3 new + 88 unchanged
- 3 fixed = 91 total (was 91) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} serde: The patch generated 142 new + 214 unchanged - 3
fixed = 356 total (was 217) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 33
unchanged - 0 fixed = 34 total (was 33) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 24 new + 1113 unchanged - 3
fixed = 1137 total (was 1116) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 50 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9618/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/patch-mvninstall-hcatalog_streaming.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/diff-checkstyle-serde.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/diff-checkstyle-hcatalog_streaming.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api serde hcatalog/streaming ql standalone-metastore U:
. |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9618/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically
[jira] [Commented] (HIVE-18872) Projection is not pushed properly when query involves multiple tables
[ https://issues.apache.org/jira/browse/HIVE-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396762#comment-16396762 ] Ankit Singhal commented on HIVE-18872: -- ping [~prasanth_j] > Projection is not pushed properly when query involves multiple tables > - > > Key: HIVE-18872 > URL: https://issues.apache.org/jira/browse/HIVE-18872 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18872.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396760#comment-16396760
]
Ankit Singhal commented on HIVE-18873:
--
ping [~prasanth_j], any help on this ticket would be greatly appreciated.
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396744#comment-16396744
]
Hive QA commented on HIVE-18908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914206/HIVE-18908.03.patch
{color:green}SUCCESS:{color} +1 due to 34 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 102 failed/errored test(s), 13434 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-18859) Incorrect handling of thrift metastore exceptions
[ https://issues.apache.org/jira/browse/HIVE-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396728#comment-16396728 ] Ganesha Shreedhara commented on HIVE-18859: --- Review request has been approved ([https://reviews.apache.org/r/65913/]). Please commit. Cc: [~akolb] > Incorrect handling of thrift metastore exceptions > - > > Key: HIVE-18859 > URL: https://issues.apache.org/jira/browse/HIVE-18859 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 2.1.1 >Reporter: Ganesha Shreedhara >Assignee: Ganesha Shreedhara >Priority: Major > Attachments: HIVE-18859.patch > > > Currently any run time exception thrown in thrift metastore during the > following operations is not getting sent to hive execution engine. > * grant/revoke role > * grant/revoke privileges > * create role > This is because ThriftHiveMetastore just handles MetaException and throws > TException during the processing of these requests. So, the command just > fails at thrift metastore end when there is run time exception (Exception can > be seen in metastore log) but the hive execution engine will keep on waiting > for the response from thrift metatstore. > > Steps to reproduce this problem : > Launch thrift metastore > Launch hive cli by passing --hiveconf > hive.metastore.uris=thrift://127.0.0.1:1 (pass the thrift metatstore host > and port) > Execute the following commands: > # set role admin > # create role test; (succeeds) > # create role test; ( hive version 2.1.1 : command is stuck, waiting for the > response from thrift metastore; hive version 1.2.1: command fails with > exception as null) > > I have uploaded the patch which has the fix in which I am handling the > checked exceptions in MetaException and throwing unchecked exceptions using > TException which fixes the problem. Please review and suggest if there is a > better way of handling this issue. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396723#comment-16396723
]
Hive QA commented on HIVE-18908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
18s{color} | {color:red} root: The patch generated 465 new + 4275 unchanged -
150 fixed = 4740 total (was 4425) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/hive-jmh: The patch generated 2 new + 16
unchanged - 1 fixed = 18 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
15s{color} | {color:red} ql: The patch generated 463 new + 3192 unchanged - 149
fixed = 3655 total (was 3341) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9617/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9617/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9617/yetus/diff-checkstyle-itests_hive-jmh.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9617/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9617/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api common serde . itests itests/hive-jmh ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9617/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add support for FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch,
> HIVE-18908.03.patch
>
>
> Currently, we do not support FULL OUTER JOIN in MapJoin.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
