[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416858#comment-16416858
]
Rui Li commented on HIVE-18831:
---
How about we register a custom serializer for {{JobResult}}, which can try to
serialize the Throwable like {{JavaSerializer}} and fall back to stack trace
string if that fails? And we can use an extra boolean to indicate whether the
Throwable is serialized, so that the deserializer can deserialize accordingly.
> Differentiate errors that are thrown by Spark tasks
> ---
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch,
> HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, HIVE-18831.7.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we
> don't differentiate between errors from HS2 / RSC vs. errors thrown by
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its
> difficult to know what part of the excution threw the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18712) Design HMS Api v2
[ https://issues.apache.org/jira/browse/HIVE-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416854#comment-16416854 ] Alexander Kolbasov commented on HIVE-18712: --- Looks like gRPC doesn't support Kerberos. > Design HMS Api v2 > - > > Key: HIVE-18712 > URL: https://issues.apache.org/jira/browse/HIVE-18712 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > This is an umbrella Jira covering the design of Hive Metastore API v2. > It is supposed to be a placeholder for discussion and design documents. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19059) Support DEFAULT keyword with INSERT and UPDATE
[
https://issues.apache.org/jira/browse/HIVE-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416823#comment-16416823
]
Hive QA commented on HIVE-19059:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 23 new + 445 unchanged - 0
fixed = 468 total (was 445) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9879/dev-support/hive-personality.sh
|
| git revision | master / 9b84ed4 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9879/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9879/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9879/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support DEFAULT keyword with INSERT and UPDATE
> --
>
> Key: HIVE-19059
> URL: https://issues.apache.org/jira/browse/HIVE-19059
> Project: Hive
> Issue Type: New Feature
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19059.1.patch
>
>
> Support DEFAULT keyword in INSERT e.g.
> {code:sql}
> INSERT INTO TABLE t values (DEFAULT, DEFAULT)
> {code}
> or with UPDATE
> {code:sql}
> UPDATE TABLE t SET col1=DEFAULT WHERE col2 > 4
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19054: - Assignee: Daniel Dai (was: Thejas M Nair) Status: Patch Available (was: Open) > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19054.1.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19054: Assignee: Thejas M Nair (was: Daniel Dai) > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Thejas M Nair >Priority: Major > Attachments: HIVE-19054.1.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416833#comment-16416833 ] Thejas M Nair commented on HIVE-19054: -- +1 > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19054.1.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19058) add object owner to HivePrivilegeObject
[
https://issues.apache.org/jira/browse/HIVE-19058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416813#comment-16416813
]
Hive QA commented on HIVE-19058:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916309/HIVE-19058.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 148 failed/errored test(s), 13425 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416805#comment-16416805 ] Sankar Hariappan commented on HIVE-19007: - Re-attached 02.patch as ptest build is not scheduled. > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch, HIVE-19007.02.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19007: Attachment: HIVE-19007.02.patch > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch, HIVE-19007.02.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19007: Status: Patch Available (was: Open) > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch, HIVE-19007.02.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19007: Attachment: (was: HIVE-19007.02.patch) > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch, HIVE-19007.02.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416803#comment-16416803 ] Sankar Hariappan commented on HIVE-18747: - Re-attached 03.patch as ptest build is not scheduled. > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, > HIVE-18747.03.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Attachment: HIVE-18747.03.patch > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, > HIVE-18747.03.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Status: Patch Available (was: Open) > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, > HIVE-18747.03.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-19007: Status: Open (was: Patch Available) > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch, HIVE-19007.02.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Attachment: (was: HIVE-18747.03.patch) > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18747: Status: Open (was: Patch Available) > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, > HIVE-18747.03.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416773#comment-16416773
]
Matt McCline commented on HIVE-19019:
-
Oh, sorry. I discovered the Complex Type bugs by executing the Map-Reduce
tests with vectorization forced ON. It turns out the normally non-vectorized
Parquet Q files with Complex Type tests triggered several execution errors
including this JIRA. No problem if you are comfortable code reading it.
Thanks.
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19058) add object owner to HivePrivilegeObject
[
https://issues.apache.org/jira/browse/HIVE-19058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416760#comment-16416760
]
Hive QA commented on HIVE-19058:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 17 new + 8 unchanged - 0 fixed
= 25 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9878/dev-support/hive-personality.sh
|
| git revision | master / 9b84ed4 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9878/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9878/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9878/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add object owner to HivePrivilegeObject
> ---
>
> Key: HIVE-19058
> URL: https://issues.apache.org/jira/browse/HIVE-19058
> Project: Hive
> Issue Type: Bug
> Components: Security
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19058.01.patch
>
>
> this can enable HiveAuthorizer to create policies based on the owner of the
> object - for example, only let the owner of a table read/write it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18197) Fix issue with wrong segments identifier usage.
[ https://issues.apache.org/jira/browse/HIVE-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra reassigned HIVE-18197: - Assignee: slim bouguerra > Fix issue with wrong segments identifier usage. > --- > > Key: HIVE-18197 > URL: https://issues.apache.org/jira/browse/HIVE-18197 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > > We have 2 different issues, that can make checking of load status fail for > druid segments. > issues are due to usage of wrong segment identifier at couple of locations. > # We are construction the segment identifier with UTC timezone, which can be > wrong if the segments we built in a different timezone. The way to fix this > is to use the segment identifier instead of re-making it at the client side. > # We are using outdate segments identifiers for the INSERT INTO case. The way > to fix this is to use the segment metadata produced by the metadata commit > phase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19052) Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of old-style MR FilterMaps
[ https://issues.apache.org/jira/browse/HIVE-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19052: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of > old-style MR FilterMaps > -- > > Key: HIVE-19052 > URL: https://issues.apache.org/jira/browse/HIVE-19052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19052.01.patch > > > Pass-Thru VectorSMBMapJoinOperator was not designed to handle old-style MR > FilterMaps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19052) Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of old-style MR FilterMaps
[ https://issues.apache.org/jira/browse/HIVE-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416743#comment-16416743 ] Matt McCline commented on HIVE-19052: - Committed to master. [~gopalv] thank you for your quick code review. > Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of > old-style MR FilterMaps > -- > > Key: HIVE-19052 > URL: https://issues.apache.org/jira/browse/HIVE-19052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19052.01.patch > > > Pass-Thru VectorSMBMapJoinOperator was not designed to handle old-style MR > FilterMaps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19052) Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of old-style MR FilterMaps
[ https://issues.apache.org/jira/browse/HIVE-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19052: Fix Version/s: 3.0.0 > Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of > old-style MR FilterMaps > -- > > Key: HIVE-19052 > URL: https://issues.apache.org/jira/browse/HIVE-19052 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19052.01.patch > > > Pass-Thru VectorSMBMapJoinOperator was not designed to handle old-style MR > FilterMaps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19057) Query result caching cannot be disabled by client
[
https://issues.apache.org/jira/browse/HIVE-19057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416742#comment-16416742
]
Hive QA commented on HIVE-19057:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916306/HIVE-19057.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13046 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18533: Attachment: HIVE-18533.8.patch > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch, > HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch, > HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18533: Attachment: (was: HIVE-18533.8.patch) > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch, > HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch, > HIVE-18533.6.patch, HIVE-18533.7.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18533: Attachment: HIVE-18533.8.patch > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch, > HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch, > HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19040) get_partitions_by_expr() implementation in HiveMetaStore causes backward incompatibility easily
[
https://issues.apache.org/jira/browse/HIVE-19040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416600#comment-16416600
]
Vihang Karajgaonkar commented on HIVE-19040:
Based on my previous discussions with [~alangates] I think the assumption with
this particular API was that only Hive uses currently and hence it is assumed
that the right hive jars are in the classpath of HMS. This API is very flaky
though from the point of view of standalone-metastore. Technically, HMS APIs
are backwards compatible which means older clients should be able to talk to
newer HMS. This assumption is broken is in this case. In my humble opinion
sending an object as bytearray over wire and deserializing it on the server
side is reinventing what thrift does for us. I am not even sure how Hive UDFs
are used to create filter strings on MySQL (I will have to look more to
understand how the expressionTree is getting generated). What happens if client
sends a UDF which is specific to Hive?
Also, is it always true that HS2 and HMS will always be at the same version?
The second meta-point to think about is when the standalone-metastore is
deployed separately, should it have hive-exec jars in its classpath? I think
what we are indirectly saying now is if Hive is one of the users of this
standalone HMS (which it most certainly will always be) then we should add the
"right" version of hive-exec jars in the metastore's classpath. How does that
make metastore standalone? Aren't we back to square one?
I think the right way ahead may be is to deprecate this API and reimplement it
without depending on hive-exec. The {{Filter.g}} is already part of HMS and we
should try to build/enhance this to provide the expressions which can be used
to filter out partitions.
> get_partitions_by_expr() implementation in HiveMetaStore causes backward
> incompatibility easily
>
>
> Key: HIVE-19040
> URL: https://issues.apache.org/jira/browse/HIVE-19040
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Priority: Major
>
> In the HiveMetaStore implementation of {{public PartitionsByExprResult
> get_partitions_by_expr(PartitionsByExprRequest req) throws TException}} , an
> expression is serialized into byte array from the client side and passed
> through PartitionsByExprRequest. Then HMS will deserialize back into the
> expression and filter the partitions by it.
> Such partition filtering expression can contain various UDFs. If there are
> some changes to one of the UDFs between different Hive versions, HS2 on the
> older version will serialize the expression in old format which won't be able
> to be deserialized by HMS on the newer version. One example of that is,
> GenericUDFIn class adds {{transient}} to the field constantInSet which will
> cause such incompatibility.
> One approach I'm thinking of is, instead of converting the expression object
> to byte array, we can pass the expression string directly.
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416595#comment-16416595 ] Sergey Shelukhin commented on HIVE-19033: - Some comments on RB. Mostly looks good, I'm just not sure how it works with ranger. See killquery for an example. > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch, > HIVE-19033.3.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19057) Query result caching cannot be disabled by client
[
https://issues.apache.org/jira/browse/HIVE-19057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416590#comment-16416590
]
Hive QA commented on HIVE-19057:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 2 new + 426 unchanged - 0
fixed = 428 total (was 426) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9877/dev-support/hive-personality.sh
|
| git revision | master / b2e411e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9877/yetus/diff-checkstyle-common.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9877/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9877/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query result caching cannot be disabled by client
> -
>
> Key: HIVE-19057
> URL: https://issues.apache.org/jira/browse/HIVE-19057
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
>Priority: Major
> Attachments: HIVE-19057.1.patch
>
>
> HIVE-18513 introduced query results caching along with some toggles to
> control enabling/disabling it. We should whiltelist the following configs so
> that the end user can dynamically control it in their session.
> {noformat}
> hive.query.results.cache.enabled
> hive.query.results.cache.wait.for.pending.results
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416589#comment-16416589 ] Sergey Shelukhin edited comment on HIVE-18052 at 3/28/18 2:02 AM: -- Attached the patch. I was assuming during initial process above that this is test only patch but there are actually some fixes and changes there. Some of these got lost because of conflicts with master in my upload. [~steveyeom2017] we urgently need to get these fixes that already exist on master... maintaining them in some floating patches is not sustainable. Also there were many out files conflicts that I've ignored (it's easier to look at files again than to resolve out file conflicts). I think at this point we should 1) Get existing fixes in. 2) Do a quick check on failures from this run and identify basic blocking issues and missing functionality. 3) Make new fixes, repeat from (1). For the first iteration we can do 1 and 2 in parallel, I'll take a look at failures from this run, or whenever we get a sane run. Throughout this process we can determine when this is shippable, ie when remaining gaps are small/trivial/obscure and we can just disable them and fix them later via the same process. Otherwise attempt to make one epic fix patch will take too long. [~steveyeom2017] also do you know if the patch/some other jira has full load overwrite implementation? I'm not sure about the state of that. cc [~ekoifman] [~hagleitn] was (Author: sershe): Attached the patch. I was assuming during initial process above that this is test only patch but there are actually some fixes and changes there. Some of these got lost because of conflicts with master in my upload. [~steveyeom2017] we urgently need to get these fixes that already exist on master... maintaining them in some floating patches is not sustainable. Also there were many out files conflicts that I've ignored (it's easier to look at files again than to resolve out file conflicts). I think at this point we should 1) Get existing fixes in. 2) Do a quick check on failures from this run and identify basic blocking issues and missing functionality. 3) Make new fixes, repeat from (1). Throughout this process we can determine when this is shippable, ie when remaining gaps are small/trivial/obscure and we can just disable them and fix them later via the same process. Otherwise attempt to make one epic fix patch will take too long. [~steveyeom2017] also do you know if the patch/some other jira has full load overwrite implementation? I'm not sure about the state of that. > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch, > HIVE-18052.6.patch, HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416589#comment-16416589 ] Sergey Shelukhin commented on HIVE-18052: - Attached the patch. I was assuming during initial process above that this is test only patch but there are actually some fixes and changes there. Some of these got lost because of conflicts with master in my upload. [~steveyeom2017] we urgently need to get these fixes that already exist on master... maintaining them in some floating patches is not sustainable. Also there were many out files conflicts that I've ignored (it's easier to look at files again than to resolve out file conflicts). I think at this point we should 1) Get existing fixes in. 2) Do a quick check on failures from this run and identify basic blocking issues and missing functionality. 3) Make new fixes, repeat from (1). Throughout this process we can determine when this is shippable, ie when remaining gaps are small/trivial/obscure and we can just disable them and fix them later via the same process. Otherwise attempt to make one epic fix patch will take too long. [~steveyeom2017] also do you know if the patch/some other jira has full load overwrite implementation? I'm not sure about the state of that. > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch, > HIVE-18052.6.patch, HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18052: --- Assignee: Sergey Shelukhin (was: Steve Yeom) > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch, > HIVE-18052.6.patch, HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18052: Attachment: HIVE-18052.18.patch > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch, > HIVE-18052.6.patch, HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18052) Run p-tests on mm tables
[ https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18052: --- Assignee: Steve Yeom (was: Sergey Shelukhin) > Run p-tests on mm tables > > > Key: HIVE-18052 > URL: https://issues.apache.org/jira/browse/HIVE-18052 > Project: Hive > Issue Type: Task >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, > HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, > HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, > HIVE-18052.17.patch, HIVE-18052.18.patch, HIVE-18052.2.patch, > HIVE-18052.3.patch, HIVE-18052.4.patch, HIVE-18052.5.patch, > HIVE-18052.6.patch, HIVE-18052.7.patch, HIVE-18052.8.patch, HIVE-18052.9.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19052) Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of old-style MR FilterMaps
[
https://issues.apache.org/jira/browse/HIVE-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416579#comment-16416579
]
Hive QA commented on HIVE-19052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916265/HIVE-19052.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13046 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Updated] (HIVE-19055) WM alter may fail if the name is not changed
[ https://issues.apache.org/jira/browse/HIVE-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19055: Attachment: HIVE-19055.01.nogen.patch > WM alter may fail if the name is not changed > > > Key: HIVE-19055 > URL: https://issues.apache.org/jira/browse/HIVE-19055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19055.01.nogen.patch, HIVE-19055.01.patch, > HIVE-19055.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19064:
---
Attachment: (was: HIVE-19064.patch)
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19064.01.patch
>
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19055) WM alter may fail if the name is not changed
[ https://issues.apache.org/jira/browse/HIVE-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416574#comment-16416574 ] Sergey Shelukhin commented on HIVE-19055: - Attached a patch and an (even smaller than the original :)) patch excluding generated code. > WM alter may fail if the name is not changed > > > Key: HIVE-19055 > URL: https://issues.apache.org/jira/browse/HIVE-19055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19055.01.nogen.patch, HIVE-19055.01.patch, > HIVE-19055.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19055) WM alter may fail if the name is not changed
[ https://issues.apache.org/jira/browse/HIVE-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19055: Attachment: HIVE-19055.01.patch > WM alter may fail if the name is not changed > > > Key: HIVE-19055 > URL: https://issues.apache.org/jira/browse/HIVE-19055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19055.01.nogen.patch, HIVE-19055.01.patch, > HIVE-19055.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19064:
---
Attachment: HIVE-19064.01.patch
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19064.01.patch
>
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19055) WM alter may fail if the name is not changed
[ https://issues.apache.org/jira/browse/HIVE-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19055: Attachment: (was: HIVE-19055.01.patch) > WM alter may fail if the name is not changed > > > Key: HIVE-19055 > URL: https://issues.apache.org/jira/browse/HIVE-19055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19055.01.nogen.patch, HIVE-19055.01.patch, > HIVE-19055.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19055) WM alter may fail if the name is not changed
[ https://issues.apache.org/jira/browse/HIVE-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416573#comment-16416573 ] Sergey Shelukhin commented on HIVE-19055: - Actually, one use case (replace active plan) does rely semantically on the null plan. Looks like local metastore doesn't go thru thrift validation so it works in tests. The better approach is to make the value nullable. > WM alter may fail if the name is not changed > > > Key: HIVE-19055 > URL: https://issues.apache.org/jira/browse/HIVE-19055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19055.01.patch, HIVE-19055.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19055) WM alter may fail if the name is not changed
[ https://issues.apache.org/jira/browse/HIVE-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19055: Attachment: HIVE-19055.01.patch > WM alter may fail if the name is not changed > > > Key: HIVE-19055 > URL: https://issues.apache.org/jira/browse/HIVE-19055 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19055.01.patch, HIVE-19055.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416559#comment-16416559
]
Vihang Karajgaonkar commented on HIVE-19019:
Hi [~mmccline] I am not super familiar with this part of the code. How is this
related to parquet complex type vectorization? Thanks!
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19064:
---
Attachment: HIVE-19064.patch
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19064.patch
>
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19064 started by Jesus Camacho Rodriguez.
--
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-19064:
---
Status: Patch Available (was: In Progress)
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-19064:
--
> Add mode to support delimited identifiers enclosed within double quotation
> --
>
> Key: HIVE-19064
> URL: https://issues.apache.org/jira/browse/HIVE-19064
> Project: Hive
> Issue Type: Improvement
> Components: Parser, SQL
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> As per SQL standard. Hive currently uses `` (backticks). Default will
> continue being backticks, but we will support identifiers within double
> quotation via configuration parameter.
> This issue will also extends support for arbitrary char sequences, e.g.,
> containing {{~ ! @ # $ % ^ & * () , < >}}, in database and table names.
> Currently, special characters are only supported for column names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19052) Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of old-style MR FilterMaps
[
https://issues.apache.org/jira/browse/HIVE-19052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416518#comment-16416518
]
Hive QA commented on HIVE-19052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 331 unchanged - 0
fixed = 335 total (was 331) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9876/dev-support/hive-personality.sh
|
| git revision | master / b2e411e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9876/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9876/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9876/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Disable Vector Pass-Thru SMB MapJoin in the presence of
> old-style MR FilterMaps
> --
>
> Key: HIVE-19052
> URL: https://issues.apache.org/jira/browse/HIVE-19052
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19052.01.patch
>
>
> Pass-Thru VectorSMBMapJoinOperator was not designed to handle old-style MR
> FilterMaps.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[ https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416510#comment-16416510 ] Sergey Shelukhin commented on HIVE-19014: - [~thejas] [~jdere] can I have a review? thanks :) > utilize YARN-8028 (queue ACL check) in Hive Tez session pool > > > Key: HIVE-19014 > URL: https://issues.apache.org/jira/browse/HIVE-19014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19014.01.patch, HIVE-19014.02.patch, > HIVE-19014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[ https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19014: Attachment: HIVE-19014.02.patch > utilize YARN-8028 (queue ACL check) in Hive Tez session pool > > > Key: HIVE-19014 > URL: https://issues.apache.org/jira/browse/HIVE-19014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19014.01.patch, HIVE-19014.02.patch, > HIVE-19014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18999) Filter operator does not work for List
[
https://issues.apache.org/jira/browse/HIVE-18999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416495#comment-16416495
]
Steve Yeom commented on HIVE-18999:
---
A better patch might be to change constant side if it is smaller in type than
the
column side since the conversion helper takes larger type as common type
at least for the test case.
> Filter operator does not work for List
> --
>
> Key: HIVE-18999
> URL: https://issues.apache.org/jira/browse/HIVE-18999
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
>
> {code:sql}
> create table table1(col0 int, col1 bigint, col2 string, col3 bigint, col4
> bigint);
> insert into table1 values (1, 1, 'ccl',2014, 11);
> insert into table1 values (1, 1, 'ccl',2015, 11);
> insert into table1 values (1, 1, 'ccl',2014, 11);
> insert into table1 values (1, 1, 'ccl',2013, 11);
> -- INCORRECT
> SELECT COUNT(t1.col0) from table1 t1 where struct(col3, col4) in
> (struct(2014,11));
> -- CORRECT
> SELECT COUNT(t1.col0) from table1 t1 where struct(col3, col4) in
> (struct('2014','11'));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18872) Projection is not pushed properly when query involves multiple tables
[ https://issues.apache.org/jira/browse/HIVE-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416496#comment-16416496 ] Prasanth Jayachandran commented on HIVE-18872: -- I don't understand the issue here. Could you please add description and repro/testcase for this issue? If the tablescan op changes the column list could also change. I am doubtful about the code removal when iterating over "dirs" (which could be from different table aliases). > Projection is not pushed properly when query involves multiple tables > - > > Key: HIVE-18872 > URL: https://issues.apache.org/jira/browse/HIVE-18872 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18872.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[
https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416487#comment-16416487
]
Sergey Shelukhin commented on HIVE-19014:
-
With some changes (will update the patch shortly)...
{noformat}
ERROR : Failed to execute tez graph.
org.apache.hadoop.hive.ql.metadata.HiveException: anonymous has no access to
dsfsdfsf
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:246)
~[hive-exec-3.0.0.3.0.0.0-1098.jar:3.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:458)
~[hive-exec-3.0.0.3.0.0.0-1098.jar:3.0.0-SNAPSHOT]
{noformat}
Seems to work ok at least on unsecure cluster
> utilize YARN-8028 (queue ACL check) in Hive Tez session pool
>
>
> Key: HIVE-19014
> URL: https://issues.apache.org/jira/browse/HIVE-19014
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19014.01.patch, HIVE-19014.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18873:
-
Status: Patch Available (was: Open)
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-18873:
Assignee: Ankit Singhal (was: Prasanth Jayachandran)
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416484#comment-16416484
]
Prasanth Jayachandran commented on HIVE-18873:
--
Sorry for lagging behind on this one. Patch looks good to me. +1, pending tests.
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result
[
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-18873:
Assignee: Prasanth Jayachandran (was: Ankit Singhal)
> Skipping predicate pushdown for MR silently at HiveInputFormat can cause
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table
> aliases,
> // since we don't clone jobConf per alias
> if (mrwork != null && mrwork.getAliases() != null &&
> mrwork.getAliases().size() > 1 &&
> jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
> return;
> }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't
> believe that predicate is handled by storage handler.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-19062: -- Status: Patch Available (was: Open) > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-19062: -- Attachment: HIVE-19062.1.patch > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17824) msck repair table should drop the missing partitions from metastore
[
https://issues.apache.org/jira/browse/HIVE-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416476#comment-16416476
]
Janaki Lahorani commented on HIVE-17824:
Hi [~sershe], [~ashutoshc],
I plan to address this by changing the syntax as follows:
MSCK [REPAIR] TABLE [ADD/DROP/SYNC PARTITIONS];
ADD is equivalent to current behavior and will be the default: The metadata
will be updated to reflect all the new partitions added in the file system.
DROP will remove the metadata corresponding to partitions that are removed from
the filesystem.
SYNC will do both ADD and DROP.
Please let us know your opinion. If I don't hear from you in the next couple
of days I will assume that you are okay with this syntax, and proceed to upload
the patch.
Thanks for your time.
> msck repair table should drop the missing partitions from metastore
> ---
>
> Key: HIVE-17824
> URL: https://issues.apache.org/jira/browse/HIVE-17824
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Janaki Lahorani
>Priority: Major
>
> {{msck repair table }} is often used in environments where the new
> partitions are loaded as directories on HDFS or S3 and users want to create
> the missing partitions in bulk. However, currently it only supports addition
> of missing partitions. If there are any partitions which are present in
> metastore but not on the FileSystem, it should also delete them so that it
> truly repairs the table metadata.
> We should be careful not to break backwards compatibility so we should either
> introduce a new config or keyword to add support to delete unnecessary
> partitions from the metastore. This way users who want the old behavior can
> easily turn it off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416466#comment-16416466
]
Sergey Shelukhin commented on HIVE-18547:
-
+1
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch, HIVE-18547.2.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416458#comment-16416458
]
Prasanth Jayachandran commented on HIVE-18547:
--
[~sershe] Since HIVE-19061 handles the update events, removed the allocation %
from test. Either GET event will have correct allocation % or there will be a
separate UPDATE event. could you take a look? small change.
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch, HIVE-18547.2.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Attachment: HIVE-18547.2.patch
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch, HIVE-18547.2.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs
[
https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416454#comment-16416454
]
Hive QA commented on HIVE-18755:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
41s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color}
| {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hcatalog/core: The patch generated 1 new + 10
unchanged - 1 fixed = 11 total (was 11) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/hcatalog-unit: The patch generated 20 new + 7
unchanged - 18 fixed = 27 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 12 new + 623
unchanged - 12 fixed = 635 total (was 635) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 18 new + 1245 unchanged - 13
fixed = 1263 total (was 1258) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} standalone-metastore: The patch generated 956 new +
2265 unchanged - 436 fixed = 3221 total (was 2701) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 154 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
4s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
8s{color} | {color:red} standalone-metastore generated 1 new + 52 unchanged - 9
fixed = 53 total (was 61) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9875/dev-support/hive-personality.sh
|
| git revision | master / f82eb8d |
| Default Java | 1.8.0_111 |
| mvninstall |
[jira] [Commented] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416445#comment-16416445
]
Gopal V commented on HIVE-18928:
Actually, I think that will hide the actual issue of two threads ending up with
the same perflogger object.
I'll keep this open until the whole of the test run is complete & I can see if
this is a symptom or the actual bug.
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19033: - Attachment: HIVE-19033.3.patch > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch, > HIVE-19033.3.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416385#comment-16416385 ] Prasanth Jayachandran edited comment on HIVE-19033 at 3/27/18 11:00 PM: Also removed cli option and web endpoint way of purging cache in the new patch. was (Author: prasanth_j): Also remove cli option and web endpoint way of purging cache. > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416430#comment-16416430 ] Ashutosh Chauhan commented on HIVE-19049: - [~nishantbangarwa] I guess you didn't rebase your patch correctly. Can you rebase it against latest master? > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18928) HS2: Perflogger has a race condition
[
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416427#comment-16416427
]
Sahil Takiar commented on HIVE-18928:
-
Ok, I think using {{ConcurrentHashMap}} is probably the safest approach then.
+1
> HS2: Perflogger has a race condition
>
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18928.1.patch
>
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196)
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342)
> ~[guava-19.0.jar:?]
> at
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327)
> ~[guava-19.0.jar:?]
> at
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218)
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
> ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416426#comment-16416426 ] Prasanth Jayachandran commented on HIVE-19033: -- RB is not responding. Will update the patch in RB as soon as its back. > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18885) DbNotificationListener has a deadlock between Java and DB locks (2.x line)
[
https://issues.apache.org/jira/browse/HIVE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18885:
---
Resolution: Fixed
Fix Version/s: 2.3.3
2.4.0
Status: Resolved (was: Patch Available)
Patch merged to branch-2 and branch-2.3 Thanks for the review [~akolb] and
[~pvary]
> DbNotificationListener has a deadlock between Java and DB locks (2.x line)
> --
>
> Key: HIVE-18885
> URL: https://issues.apache.org/jira/browse/HIVE-18885
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 2.3.2
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.4.0, 2.3.3
>
> Attachments: HIVE-18885.01.branch-2.patch,
> HIVE-18885.02.branch-2.patch, HIVE-18885.03-branch-2.patch,
> HIVE-18885.04-branch-2.patch
>
>
> You can see the problem from looking at the code, but it actually created
> severe problems for real life Hive user.
> When {{alter table}} has {{cascade}} option it does the following:
> {code:java}
> msdb.openTransaction()
> ...
> List parts = msdb.getPartitions(dbname, name, -1);
> for (Partition part : parts) {
> List oldCols = part.getSd().getCols();
> part.getSd().setCols(newt.getSd().getCols());
> String oldPartName =
> Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
> updatePartColumnStatsForAlterColumns(msdb, part, oldPartName,
> part.getValues(), oldCols, part);
> msdb.alterPartition(dbname, name, part.getValues(), part);
> }
> {code}
> So it walks all partitions (and this may be huge list) and does some
> non-trivial operations in one single uber-transaction.
> When DbNotificationListener is enabled, it adds an event for each partition,
> all while
> holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is
> happening no other write DDL can proceed. This can sometimes cause DB lock
> timeouts which cause HMS level operation retries which make things even worse.
> In one particular case this pretty much made HMS unusable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19033: - Attachment: HIVE-19033.2.patch > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416385#comment-16416385 ] Prasanth Jayachandran commented on HIVE-19033: -- Also remove cli option and web endpoint way of purging cache. > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18994) Handle client connections on failover
[ https://issues.apache.org/jira/browse/HIVE-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18994: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) TestActivePassiveHA test failure seems fixed. Other test failures are unrelated. Committed patch to master. Thanks for the reviews! > Handle client connections on failover > - > > Key: HIVE-18994 > URL: https://issues.apache.org/jira/browse/HIVE-18994 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18994.1.patch, HIVE-18994.2.patch, > HIVE-18994.3.patch > > > When leader failover happens (either automatically or manually), tez sessions > are closed. But client connections are not. We need to close the client > connections explicitly so that workload manager revokes all the guaranteed > slots and upon reconnection client will connect to active HS2 instance (this > is to avoid clients reusing the same connection and submitting queries to > passive HS2). In future, some timeout or other policies (may be WM will run > everything speculatively) can be added. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19050) DBNotificationListener does not catch exceptions in the cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416363#comment-16416363
]
Vihang Karajgaonkar commented on HIVE-19050:
Adding a RB link. [~akolb] [~pvary] can you take a look?
> DBNotificationListener does not catch exceptions in the cleaner thread
> --
>
> Key: HIVE-19050
> URL: https://issues.apache.org/jira/browse/HIVE-19050
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-19050.01.patch
>
>
> The DbNotificationListener class has a separate thread which cleans the old
> notifications from the database. Here is the snippet from the {{run}} method.
> {noformat}
> public void run() {
> while (true) {
> rs.cleanNotificationEvents(ttl);
> LOG.debug("Cleaner thread done");
> try {
> Thread.sleep(sleepTime);
> } catch (InterruptedException e) {
> LOG.info("Cleaner thread sleep interrupted", e);
> }
> }
> }
> {noformat}
> If {{rs.cleanNotificationEvents}} throws a RuntimeException which datanucleus
> can throw the exception remains uncaught and the thread will die. This can
> lead to older notifications never getting cleaned until we restart HMS. Given
> that many operations generate loads of events, the notification log table can
> quickly have thousands of rows which are never get cleaned up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19050) DBNotificationListener does not catch exceptions in the cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19050:
---
Attachment: (was: HIVE-19050.01.patch)
> DBNotificationListener does not catch exceptions in the cleaner thread
> --
>
> Key: HIVE-19050
> URL: https://issues.apache.org/jira/browse/HIVE-19050
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-19050.01.patch
>
>
> The DbNotificationListener class has a separate thread which cleans the old
> notifications from the database. Here is the snippet from the {{run}} method.
> {noformat}
> public void run() {
> while (true) {
> rs.cleanNotificationEvents(ttl);
> LOG.debug("Cleaner thread done");
> try {
> Thread.sleep(sleepTime);
> } catch (InterruptedException e) {
> LOG.info("Cleaner thread sleep interrupted", e);
> }
> }
> }
> {noformat}
> If {{rs.cleanNotificationEvents}} throws a RuntimeException which datanucleus
> can throw the exception remains uncaught and the thread will die. This can
> lead to older notifications never getting cleaned until we restart HMS. Given
> that many operations generate loads of events, the notification log table can
> quickly have thousands of rows which are never get cleaned up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19050) DBNotificationListener does not catch exceptions in the cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19050:
---
Attachment: HIVE-19050.01.patch
> DBNotificationListener does not catch exceptions in the cleaner thread
> --
>
> Key: HIVE-19050
> URL: https://issues.apache.org/jira/browse/HIVE-19050
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-19050.01.patch
>
>
> The DbNotificationListener class has a separate thread which cleans the old
> notifications from the database. Here is the snippet from the {{run}} method.
> {noformat}
> public void run() {
> while (true) {
> rs.cleanNotificationEvents(ttl);
> LOG.debug("Cleaner thread done");
> try {
> Thread.sleep(sleepTime);
> } catch (InterruptedException e) {
> LOG.info("Cleaner thread sleep interrupted", e);
> }
> }
> }
> {noformat}
> If {{rs.cleanNotificationEvents}} throws a RuntimeException which datanucleus
> can throw the exception remains uncaught and the thread will die. This can
> lead to older notifications never getting cleaned until we restart HMS. Given
> that many operations generate loads of events, the notification log table can
> quickly have thousands of rows which are never get cleaned up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19049: Attachment: HIVE-19049.02.patch > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416357#comment-16416357 ] Nishant Bangarwa commented on HIVE-19049: - [~ashutoshc] done with review comments, please check. > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18994) Handle client connections on failover
[
https://issues.apache.org/jira/browse/HIVE-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416354#comment-16416354
]
Hive QA commented on HIVE-18994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916293/HIVE-18994.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 166 failed/errored test(s), 13817 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=104)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[constraint_partition_columns]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_convert]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_truncate_cols]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[show_tables_bad_db1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=95)
[jira] [Updated] (HIVE-19050) DBNotificationListener does not catch exceptions in the cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19050:
---
Status: Patch Available (was: Open)
> DBNotificationListener does not catch exceptions in the cleaner thread
> --
>
> Key: HIVE-19050
> URL: https://issues.apache.org/jira/browse/HIVE-19050
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-19050.01.patch
>
>
> The DbNotificationListener class has a separate thread which cleans the old
> notifications from the database. Here is the snippet from the {{run}} method.
> {noformat}
> public void run() {
> while (true) {
> rs.cleanNotificationEvents(ttl);
> LOG.debug("Cleaner thread done");
> try {
> Thread.sleep(sleepTime);
> } catch (InterruptedException e) {
> LOG.info("Cleaner thread sleep interrupted", e);
> }
> }
> }
> {noformat}
> If {{rs.cleanNotificationEvents}} throws a RuntimeException which datanucleus
> can throw the exception remains uncaught and the thread will die. This can
> lead to older notifications never getting cleaned until we restart HMS. Given
> that many operations generate loads of events, the notification log table can
> quickly have thousands of rows which are never get cleaned up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19050) DBNotificationListener does not catch exceptions in the cleaner thread
[
https://issues.apache.org/jira/browse/HIVE-19050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-19050:
---
Attachment: HIVE-19050.01.patch
> DBNotificationListener does not catch exceptions in the cleaner thread
> --
>
> Key: HIVE-19050
> URL: https://issues.apache.org/jira/browse/HIVE-19050
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-19050.01.patch
>
>
> The DbNotificationListener class has a separate thread which cleans the old
> notifications from the database. Here is the snippet from the {{run}} method.
> {noformat}
> public void run() {
> while (true) {
> rs.cleanNotificationEvents(ttl);
> LOG.debug("Cleaner thread done");
> try {
> Thread.sleep(sleepTime);
> } catch (InterruptedException e) {
> LOG.info("Cleaner thread sleep interrupted", e);
> }
> }
> }
> {noformat}
> If {{rs.cleanNotificationEvents}} throws a RuntimeException which datanucleus
> can throw the exception remains uncaught and the thread will die. This can
> lead to older notifications never getting cleaned until we restart HMS. Given
> that many operations generate loads of events, the notification log table can
> quickly have thousands of rows which are never get cleaned up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19056) IllegalArgumentException in FixAcidKeyIndex when ORC file has 0 rows
[
https://issues.apache.org/jira/browse/HIVE-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-19056:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master
> IllegalArgumentException in FixAcidKeyIndex when ORC file has 0 rows
>
>
> Key: HIVE-19056
> URL: https://issues.apache.org/jira/browse/HIVE-19056
> Project: Hive
> Issue Type: Bug
> Components: ORC, Transactions
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19056.1.patch
>
>
> {noformat}
> ERROR recovering
> /Users/jdere/dev/hwx/gerrit/hive2-gerrit/ql/target/tmp/TestFixAcidKeyIndex.testValidKeyIndex.orc
> java.lang.IllegalArgumentException: Seek to a negative row number -1
> at
> org.apache.orc.impl.RecordReaderImpl.seekToRow(RecordReaderImpl.java:1300)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.seekToRow(RecordReaderImpl.java:101)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.recoverFile(FixAcidKeyIndex.java:232)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.recoverFiles(FixAcidKeyIndex.java:132)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.main(FixAcidKeyIndex.java:104)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19033) Provide an option to purge LLAP IO cache
[
https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416312#comment-16416312
]
Prasanth Jayachandran edited comment on HIVE-19033 at 3/27/18 10:18 PM:
Based on offline feedback from [~gopalv] added llap specific commands under
HiveCommandOperation to get Hive authorization. Also updated the patch to get
use LLAP management API for security.
2 commands are add
1) LLAP cluster commands (only info is implemented in this patch)
2) LLAP cache commands (only purge is implemented in this patch)
"llap cluster -info;" has no authorization, any user can read this information.
"llap cache -purge;" requires users to have admin role.
Some examples:
{code:title=user in non-admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
+---+
|set|
+---+
| hive.security.authorization.enabled=true |
+---+
1 row selected (0.165 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
Error: Error while processing statement: Permission denied: Principal
[name=pjayachandran, type=USER] does not have following privileges for
operation LLAP_CACHE [[ADMIN PRIVILEGE] on Object [type=COMMAND_PARAMS,
name=[-purge]]] (state=,code=1)
{code}
{code:title=user in admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set role admin;
No rows affected (1.019 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
+++
| hostName | purgedMemoryBytes |
+++
| localhost | 50429952 |
+++
{code}
{code:title=when authZ is disabled, user can read cluster info}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
++
|set |
++
| hive.security.authorization.enabled=false |
++
1 row selected (0.159 seconds)
0: jdbc:hive2://localhost:1> llap cluster -info;
++---++--+-+-+
| applicationId |workerIdentity | hostname | rpcPort
| memory| vcores |
++---++--+-+-+
| null | 873b7438-01b1-4974-90e2-1c5631602db9 | localhost | 15001
| 3145728000 | 3 |
++---++--+-+-+
{code}
ignore "null" applicationId as this is my local setup which is not deployed via
slider or yarn services.
was (Author: prasanth_j):
Based on offline feedback from [~gopalv] added llap specific commands under
HiveCommandOperation to get Hive authorization. Also updated the patch to get
use LLAP management API for security.
2 commands are add
1) LLAP cluster commands (only info is implemented in this patch)
2) LLAP cache commands (only purge is implemented in this patch)
"llap cluster -info;" has no authorization, any user can read this information.
"llap cache -purge;" requires users to have admin role.
Some examples:
{code:title=user in non-admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
+---+
|set|
+---+
| hive.security.authorization.enabled=true |
+---+
1 row selected (0.165 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
Error: Error while processing statement: Permission denied: Principal
[name=pjayachandran, type=USER] does not have following privileges for
operation LLAP_CACHE_PURGE [ADMIN PRIVILEGE on INPUT] (state=,code=1)
{code}
{code:title=user in admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set role admin;
No rows affected (1.019 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
+++
| hostName | purgedMemoryBytes |
+++
| localhost | 50429952 |
+++
{code}
{code:title=when authZ is disabled, user can read cluster info}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
++
|set |
++
| hive.security.authorization.enabled=false |
++
1 row selected (0.159 seconds)
0:
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[
https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416312#comment-16416312
]
Prasanth Jayachandran commented on HIVE-19033:
--
Based on offline feedback from [~gopalv] added llap specific commands under
HiveCommandOperation to get Hive authorization. Also updated the patch to get
use LLAP management API for security.
2 commands are add
1) LLAP cluster commands (only info is implemented in this patch)
2) LLAP cache commands (only purge is implemented in this patch)
"llap cluster -info;" has no authorization, any user can read this information.
"llap cache -purge;" requires users to have admin role.
Some examples:
{code:title=user in non-admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
+---+
|set|
+---+
| hive.security.authorization.enabled=true |
+---+
1 row selected (0.165 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
Error: Error while processing statement: Permission denied: Principal
[name=pjayachandran, type=USER] does not have following privileges for
operation LLAP_CACHE_PURGE [ADMIN PRIVILEGE on INPUT] (state=,code=1)
{code}
{code:title=user in admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set role admin;
No rows affected (1.019 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
+++
| hostName | purgedMemoryBytes |
+++
| localhost | 50429952 |
+++
{code}
{code:title=when authZ is disabled, user can read cluster info}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
++
|set |
++
| hive.security.authorization.enabled=false |
++
1 row selected (0.159 seconds)
0: jdbc:hive2://localhost:1> llap cluster -info;
++---++--+-+-+
| applicationId |workerIdentity | hostname | rpcPort
| memory| vcores |
++---++--+-+-+
| null | 873b7438-01b1-4974-90e2-1c5631602db9 | localhost | 15001
| 3145728000 | 3 |
++---++--+-+-+
{code}
ignore "null" applicationId as this is my local setup which is not deployed via
slider or yarn services.
> Provide an option to purge LLAP IO cache
>
>
> Key: HIVE-19033
> URL: https://issues.apache.org/jira/browse/HIVE-19033
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19033.1.patch
>
>
> Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI
> tool to invoke the endpoint of all LLAP daemons.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-18982) Provide a CLI option to manually trigger failover
[
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18982:
-
Comment: was deleted
(was: Based on offline feedback from [~gopalv] added llap specific commands
under HiveCommandOperation to get Hive authorization. Also updated the patch to
get use LLAP management API for security.
2 commands are add
1) LLAP cluster commands (only info is implemented in this patch)
2) LLAP cache commands (only purge is implemented in this patch)
"llap cluster -info;" has no authorization, any user can read this information.
"llap cache -purge;" requires users to have admin role.
Some examples:
{code:title=user in non-admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
+---+
|set|
+---+
| hive.security.authorization.enabled=true |
+---+
1 row selected (0.165 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
Error: Error while processing statement: Permission denied: Principal
[name=pjayachandran, type=USER] does not have following privileges for
operation LLAP_CACHE_PURGE [ADMIN PRIVILEGE on INPUT] (state=,code=1)
{code}
{code:title=user in admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set role admin;
No rows affected (1.019 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
+++
| hostName | purgedMemoryBytes |
+++
| localhost | 50429952 |
+++
{code}
{code:title=when authZ is disabled, user can read cluster info}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
++
|set |
++
| hive.security.authorization.enabled=false |
++
1 row selected (0.159 seconds)
0: jdbc:hive2://localhost:1> llap cluster -info;
++---++--+-+-+
| applicationId |workerIdentity | hostname | rpcPort
| memory| vcores |
++---++--+-+-+
| null | 873b7438-01b1-4974-90e2-1c5631602db9 | localhost | 15001
| 3145728000 | 3 |
++---++--+-+-+
{code}
ignore "null" applicationId as this is my local setup which is not deployed via
slider or yarn services. )
> Provide a CLI option to manually trigger failover
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18982.1.patch, HIVE-18982.2.patch,
> HIVE-18982.3.patch, HIVE-18982.4.patch, HIVE-18982.5.patch
>
>
> HIVE-18281 added active-passive HA. There might be a administrative need to
> trigger a manual failover of HS2 Active server. Add command line tool to view
> list of all HS2 instances and trigger manual failover (only under force
> mode). The clients currently connected to active HS2 will be closed. In
> future, more options to existing clients connections can be handled via
> configs/options (like wait until timeout, wait until current sessions are
> closed etc.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18982) Provide a CLI option to manually trigger failover
[
https://issues.apache.org/jira/browse/HIVE-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416308#comment-16416308
]
Prasanth Jayachandran commented on HIVE-18982:
--
Based on offline feedback from [~gopalv] added llap specific commands under
HiveCommandOperation to get Hive authorization. Also updated the patch to get
use LLAP management API for security.
2 commands are add
1) LLAP cluster commands (only info is implemented in this patch)
2) LLAP cache commands (only purge is implemented in this patch)
"llap cluster -info;" has no authorization, any user can read this information.
"llap cache -purge;" requires users to have admin role.
Some examples:
{code:title=user in non-admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
+---+
|set|
+---+
| hive.security.authorization.enabled=true |
+---+
1 row selected (0.165 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
Error: Error while processing statement: Permission denied: Principal
[name=pjayachandran, type=USER] does not have following privileges for
operation LLAP_CACHE_PURGE [ADMIN PRIVILEGE on INPUT] (state=,code=1)
{code}
{code:title=user in admin role trying to purge the cache}
0: jdbc:hive2://localhost:1> set role admin;
No rows affected (1.019 seconds)
0: jdbc:hive2://localhost:1> llap cache -purge;
+++
| hostName | purgedMemoryBytes |
+++
| localhost | 50429952 |
+++
{code}
{code:title=when authZ is disabled, user can read cluster info}
0: jdbc:hive2://localhost:1> set hive.security.authorization.enabled;
++
|set |
++
| hive.security.authorization.enabled=false |
++
1 row selected (0.159 seconds)
0: jdbc:hive2://localhost:1> llap cluster -info;
++---++--+-+-+
| applicationId |workerIdentity | hostname | rpcPort
| memory| vcores |
++---++--+-+-+
| null | 873b7438-01b1-4974-90e2-1c5631602db9 | localhost | 15001
| 3145728000 | 3 |
++---++--+-+-+
{code}
ignore "null" applicationId as this is my local setup which is not deployed via
slider or yarn services.
> Provide a CLI option to manually trigger failover
> -
>
> Key: HIVE-18982
> URL: https://issues.apache.org/jira/browse/HIVE-18982
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18982.1.patch, HIVE-18982.2.patch,
> HIVE-18982.3.patch, HIVE-18982.4.patch, HIVE-18982.5.patch
>
>
> HIVE-18281 added active-passive HA. There might be a administrative need to
> trigger a manual failover of HS2 Active server. Add command line tool to view
> list of all HS2 instances and trigger manual failover (only under force
> mode). The clients currently connected to active HS2 will be closed. In
> future, more options to existing clients connections can be handled via
> configs/options (like wait until timeout, wait until current sessions are
> closed etc.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-18986: Status: Patch Available (was: In Progress) Reattach patch-3: the test failures don't seem to be related. Tries some of them which also fail without the patch. > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-18986: Attachment: HIVE-18986.3.patch > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-18986: Attachment: (was: HIVE-18986.3.patch) > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-18986: Status: In Progress (was: Patch Available) > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18702) INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
[
https://issues.apache.org/jira/browse/HIVE-18702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416291#comment-16416291
]
Ashutosh Chauhan commented on HIVE-18702:
-
[~osayankin] Can you reupload the patch and mark it Patch Available so that
Hive QA can run on it ?
> INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
> ---
>
> Key: HIVE-18702
> URL: https://issues.apache.org/jira/browse/HIVE-18702
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 3.0.0, 2.3.3
>
> Attachments: HIVE-18702.1.patch, HIVE-18702.2.patch
>
>
> Enable Hive on TEZ. (MR works fine).
> *STEP 1. Create test data*
> {code}
> nano /home/test/users.txt
> {code}
> Add to file:
> {code}
> Peter,34
> John,25
> Mary,28
> {code}
> {code}
> hadoop fs -mkdir /bug
> hadoop fs -copyFromLocal /home/test/users.txt /bug
> hadoop fs -ls /bug
> {code}
> *EXPECTED RESULT:*
> {code}
> Found 2 items
>
> -rwxr-xr-x 3 root root 25 2015-10-15 16:11 /bug/users.txt
> {code}
> *STEP 2. Upload data to hive*
> {code}
> create external table bug(name string, age int) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',' LINES TERMINATED BY '\n' LOCATION '/bug';
> select * from bug;
> {code}
> *EXPECTED RESULT:*
> {code}
> OK
> Peter 34
> John25
> Mary28
> {code}
> {code}
> create external table bug1(name string, age int) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ',' LINES TERMINATED BY '\n' LOCATION '/bug1';
> insert overwrite table bug select * from bug1;
> select * from bug;
> {code}
> *EXPECTED RESULT:*
> {code}
> OK
> Time taken: 0.097 seconds
> {code}
> *ACTUAL RESULT:*
> {code}
> hive> select * from bug;
> OK
> Peter 34
> John 25
> Mary 28
> Time taken: 0.198 seconds, Fetched: 3 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18994) Handle client connections on failover
[
https://issues.apache.org/jira/browse/HIVE-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416285#comment-16416285
]
Hive QA commented on HIVE-18994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
35s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} util in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
42s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 5 new + 14
unchanged - 0 fixed = 19 total (was 14) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 42 unchanged -
4 fixed = 42 total (was 46) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9874/dev-support/hive-personality.sh
|
| git revision | master / 245c39b |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus/patch-mvninstall-itests_hive-unit.txt
|
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus/patch-mvninstall-itests_util.txt
|
| compile |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus/patch-compile-itests_hive-unit.txt
|
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus/patch-compile-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hive-unit itests/util service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9874/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Handle client connections on failover
> -
>
> Key: HIVE-18994
> URL: https://issues.apache.org/jira/browse/HIVE-18994
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter:
[jira] [Commented] (HIVE-19047) Only the first init file is interpreted
[
https://issues.apache.org/jira/browse/HIVE-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416284#comment-16416284
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19047:
-
Adding unit test. Also made changes to the runInit method so that it works
according to the --force option.
> Only the first init file is interpreted
> ---
>
> Key: HIVE-19047
> URL: https://issues.apache.org/jira/browse/HIVE-19047
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19047.1.patch, HIVE-19047.2.patch
>
>
> I've passed multiple {{-i}} options to beeline; and I've expected to load
> both of them...but unfortunately it only parsed the first file (and ignored
> the second entirely)
> I think it would be better to:
> * either reject the command if it has multiple "-i" given
> * or load *all* "-i" scripts
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19047) Only the first init file is interpreted
[
https://issues.apache.org/jira/browse/HIVE-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-19047:
Attachment: HIVE-19047.2.patch
> Only the first init file is interpreted
> ---
>
> Key: HIVE-19047
> URL: https://issues.apache.org/jira/browse/HIVE-19047
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19047.1.patch, HIVE-19047.2.patch
>
>
> I've passed multiple {{-i}} options to beeline; and I've expected to load
> both of them...but unfortunately it only parsed the first file (and ignored
> the second entirely)
> I think it would be better to:
> * either reject the command if it has multiple "-i" given
> * or load *all* "-i" scripts
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19048) Initscript errors are ignored
[
https://issues.apache.org/jira/browse/HIVE-19048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416278#comment-16416278
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19048:
-
I checked the code, and this seems to be the default behavior.
Also, there is a --force option in the [Beeline
commands|https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommands],
which can be used to continue execution even after an error in the initFile.
> Initscript errors are ignored
> -
>
> Key: HIVE-19048
> URL: https://issues.apache.org/jira/browse/HIVE-19048
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> I've been running some queries for a while when I've noticed that my
> initscript has an error; and beeline stops interpreting the initscript after
> encountering the first error.
> {code}
> echo 'invalid;' > init.sql
> echo 'select 1;' > s1.sql
> beeline -u jdbc:hive2://localhost:1/ -n hive -i init.sql -f s1.sql
> [...]
> Running init script init.sql
> 0: jdbc:hive2://localhost:1/> invalid;
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'invalid' '' '' (state=42000,code=4)
> 0: jdbc:hive2://localhost:1/> select 1;
> [...]
> $ echo $?
> 0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19048) Initscript errors are ignored
[
https://issues.apache.org/jira/browse/HIVE-19048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19048 started by Bharathkrishna Guruvayoor Murali.
---
> Initscript errors are ignored
> -
>
> Key: HIVE-19048
> URL: https://issues.apache.org/jira/browse/HIVE-19048
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> I've been running some queries for a while when I've noticed that my
> initscript has an error; and beeline stops interpreting the initscript after
> encountering the first error.
> {code}
> echo 'invalid;' > init.sql
> echo 'select 1;' > s1.sql
> beeline -u jdbc:hive2://localhost:1/ -n hive -i init.sql -f s1.sql
> [...]
> Running init script init.sql
> 0: jdbc:hive2://localhost:1/> invalid;
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'invalid' '' '' (state=42000,code=4)
> 0: jdbc:hive2://localhost:1/> select 1;
> [...]
> $ echo $?
> 0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19056) IllegalArgumentException in FixAcidKeyIndex when ORC file has 0 rows
[
https://issues.apache.org/jira/browse/HIVE-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416257#comment-16416257
]
Hive QA commented on HIVE-19056:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916290/HIVE-19056.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 13045 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
