[jira] [Commented] (HIVE-16144) CompactionInfo doesn't have equals/hashCode but used in Set

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461668#comment-16461668
 ] 

Eugene Koifman commented on HIVE-16144:
---

I don't think CompactionInfo is used as a key anywhere so it's not critical but 
maybe good hygiene


> CompactionInfo doesn't have equals/hashCode but used in Set
> ---
>
> Key: HIVE-16144
> URL: https://issues.apache.org/jira/browse/HIVE-16144
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-16144.01.patch, HIVE-16144.02.patch
>
>
> CompactionTxnHandler.findPotentialCompactions() uses a Set 
> but CompactionInfo doesn't have equals/hashCode.
> should do the same as CompactionInfo.compareTo()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18131) Truncate table for Acid tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461553#comment-16461553
 ] 

Eugene Koifman commented on HIVE-18131:
---

the patch only adds a test to make sure we acquire X lock for acid tables.  
changing this to create a new base_x will be done in 3.1 ideally after we add a 
new X_write lock that blocks other writes but not reads.

> Truncate table for Acid tables
> --
>
> Key: HIVE-18131
> URL: https://issues.apache.org/jira/browse/HIVE-18131
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18131.01.patch
>
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-18131) Truncate table for Acid tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461553#comment-16461553
 ] 

Eugene Koifman edited comment on HIVE-18131 at 5/2/18 7:56 PM:
---

the patch only adds a test to make sure we acquire X lock for acid tables.  
changing this to create a new base_x will be done in 3.1 ideally after we add a 
new X_write lock that blocks other writes but not reads.

[~prasanth_j] could you review please


was (Author: ekoifman):
the patch only adds a test to make sure we acquire X lock for acid tables.  
changing this to create a new base_x will be done in 3.1 ideally after we add a 
new X_write lock that blocks other writes but not reads.

> Truncate table for Acid tables
> --
>
> Key: HIVE-18131
> URL: https://issues.apache.org/jira/browse/HIVE-18131
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18131.01.patch
>
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19387) CLONE - Truncate table for Acid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19387:
--
Target Version/s: 3.1.0  (was: 3.0.0)

> CLONE - Truncate table for Acid tables
> --
>
> Key: HIVE-19387
> URL: https://issues.apache.org/jira/browse/HIVE-19387
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19387) CLONE - Truncate table for Acid tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461552#comment-16461552
 ] 

Eugene Koifman commented on HIVE-19387:
---

from HIVE-18131, X locks is acquired.
For 3.1, should make this use IOW and create an empty base_x.

> CLONE - Truncate table for Acid tables
> --
>
> Key: HIVE-19387
> URL: https://issues.apache.org/jira/browse/HIVE-19387
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18131) Truncate table for Acid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18131:
--
Attachment: HIVE-18131.01.patch

> Truncate table for Acid tables
> --
>
> Key: HIVE-18131
> URL: https://issues.apache.org/jira/browse/HIVE-18131
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18131.01.patch
>
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19387) CLONE - Truncate table for Acid tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-19387:
-


> CLONE - Truncate table for Acid tables
> --
>
> Key: HIVE-19387
> URL: https://issues.apache.org/jira/browse/HIVE-19387
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.03-branch-3.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.01-branch-3.patch, 
> HIVE-19322.02-branch-3.patch, HIVE-19322.02.patch, 
> HIVE-19322.03-branch-3.patch, HIVE-19322.03.patch, HIVE-19322.04.patch, 
> HIVE-19322.05.patch, HIVE-19322.06.patch, HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18570:
--
Attachment: HIVE-18570.04-branch-3.patch

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, 
> HIVE-18570.02-branch-3.patch, HIVE-18570.03-branch-3.patch, 
> HIVE-18570.04-branch-3.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17457:
--
Attachment: HIVE-17457.03.patch

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17457.01.patch, HIVE-17457.02.patch, 
> HIVE-17457.03.patch
>
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17457:
--
Attachment: HIVE-17457.02.patch

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17457.01.patch, HIVE-17457.02.patch
>
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18570:
--
Attachment: HIVE-18570.03-branch-3.patch

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, 
> HIVE-18570.02-branch-3.patch, HIVE-18570.03-branch-3.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.02-branch-3.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.01-branch-3.patch, 
> HIVE-19322.02-branch-3.patch, HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch, HIVE-19322.06.patch, 
> HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19211) New streaming ingest API and support for dynamic partitioning

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460200#comment-16460200
 ] 

Eugene Koifman commented on HIVE-19211:
---

+1 pending tests patch 12

> New streaming ingest API and support for dynamic partitioning
> -
>
> Key: HIVE-19211
> URL: https://issues.apache.org/jira/browse/HIVE-19211
> Project: Hive
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19211.1.patch, HIVE-19211.10.patch, 
> HIVE-19211.11.patch, HIVE-19211.12.patch, HIVE-19211.2.patch, 
> HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, 
> HIVE-19211.6.patch, HIVE-19211.7.patch, HIVE-19211.8.patch, HIVE-19211.9.patch
>
>
> - New streaming API under new hive sub-module
> - Dynamic partitioning support
> - Auto-rollover transactions
> - Automatic heartbeating



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459921#comment-16459921
 ] 

Eugene Koifman commented on HIVE-19369:
---

if you want readers to die rather than wait you can just set the lock 
acquisition retry count to 0



> Locks: Add new lock implementations for always zero-wait readers
> 
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
>  Issue Type: Improvement
>Reporter: Gopal V
>Priority: Major
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking 
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to 
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to 
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an 
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different 
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending 
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, 
> there's no semantic reason to wait for them to succeed before going ahead 
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete 
> description.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459916#comment-16459916
 ] 

Eugene Koifman commented on HIVE-18570:
---

patch 2 fixes TestDbTxnManager2.testShowTablesLock which was not caused by 
these changes but was surfaced by them (the drop table command at start of test 
was wrong).

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, 
> HIVE-18570.02-branch-3.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18570:
--
Attachment: HIVE-18570.02-branch-3.patch

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch, 
> HIVE-18570.02-branch-3.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19375) "'transactional'='false' is no longer a valid property and will be ignored:

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-19375:
-


> "'transactional'='false' is no longer a valid property and will be ignored: 
> 
>
> Key: HIVE-19375
> URL: https://issues.apache.org/jira/browse/HIVE-19375
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> from {{TransactionalValidationListener.handleCreateTableTransactionalProp()}}
> {noformat}
> if ("false".equalsIgnoreCase(transactional)) {
>   // just drop transactional=false.  For backward compatibility in case 
> someone has scripts
>   // with transactional=false
>   LOG.info("'transactional'='false' is no longer a valid property and 
> will be ignored: " +
> Warehouse.getQualifiedName(newTable));
>   return;
> }
> {noformat}
> this msg is misleading since with metastore.create.as.acid=true, setting 
> transactional=false is valid to make a flat table



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459868#comment-16459868
 ] 

Eugene Koifman commented on HIVE-18570:
---

[~gopalv]
longer term we should should support snapshot isolation with optimistic 
concurrency (as much as possible) and read committed with pessimistic lock 
based concurrency.  Optimistic conflict detection currently only tracks 
update/delete conflicts and making it track inserts as well is a bigger change 
than I want to do for 3.0

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18570:
--
Attachment: HIVE-18570.01-branch-3.patch

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01-branch-3.patch, HIVE-18570.01.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.01-branch-3.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.01-branch-3.patch, HIVE-19322.02.patch, 
> HIVE-19322.03.patch, HIVE-19322.04.patch, HIVE-19322.05.patch, 
> HIVE-19322.06.patch, HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459359#comment-16459359
 ] 

Eugene Koifman commented on HIVE-19322:
---

reran some failed tests locally - all pass
committed to matser

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch, HIVE-19322.06.patch, 
> HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459334#comment-16459334
 ] 

Eugene Koifman commented on HIVE-17457:
---

the patch just adds a test for this - behavior was already correct (since 
HIVE-18589 )
[~sershe] could you review please

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17457.01.patch
>
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17457:
--
Attachment: HIVE-17457.01.patch

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17457.01.patch
>
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17457:
--
Status: Patch Available  (was: Open)

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17457.01.patch
>
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18154) IOW Acid Load Data/Insert with Overwrite in multi statement transactions

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459309#comment-16459309
 ] 

Eugene Koifman commented on HIVE-18154:
---

same for MM tables

> IOW Acid Load Data/Insert with Overwrite in multi statement transactions
> 
>
> Key: HIVE-18154
> URL: https://issues.apache.org/jira/browse/HIVE-18154
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> Consider:
> {noformat}
> START TRANSACTION
> insert into T values(1,2),(3,4)
> load data local inpath '" + getWarehouseDir() + "/1/data' overwrite into 
> table T
> update T set a = 0 where a = 6
> COMMIT
> {noformat}
> So what we should have on disk is
> {noformat}
> ├── base_028
> │   ├── 00_0
> │   └── _metadata_acid
> ├── delete_delta_028_028_0002
> │   └── bucket_0
> ├── delta_028_028_
> │   └── bucket_0
> └── delta_028_028_0002
> └── bucket_0
> {noformat}
> where base_28 is from overwrite, delta_028_028_ from 1st insert 
> nad delta_028_028_0002/delete_delta_028_028_0002 is from 
> update.
> AcidUtils.getAcidState() only returns base_28 thinking that all other deltas 
> are included in it - not what we want here.  
> Same applies for Insert Overwrite.
> The simple way to get correct behavior is to disallow commands with Overwrite 
> clause in multi-statement txns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459287#comment-16459287
 ] 

Eugene Koifman commented on HIVE-18570:
---

not for 3.0, but longer term yes

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459256#comment-16459256
 ] 

Eugene Koifman commented on HIVE-18570:
---

[~sershe] could you review please

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18570:
--
Status: Patch Available  (was: Open)

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18570:
--
Attachment: HIVE-18570.01.patch

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18570.01.patch
>
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17457:
--
Attachment: (was: HIVE-17457.01.patch)

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17457:
--
Attachment: HIVE-17457.01.patch

> IOW Acid Insert Overwrite when the transaction fails
> 
>
> Key: HIVE-17457
> URL: https://issues.apache.org/jira/browse/HIVE-17457
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-17457.01.patch
>
>
> HIVE-14988 adds support for Insert Overwrite for Acid tables.
> once we have direct write to target dir (i.e. no move op) - how do we handle 
> the case where the txn running IOW aborts?  See if getAcidState() does the 
> right thing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17660) Compaction for MM runs Cleaner - needs test once IOW is supported

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459047#comment-16459047
 ] 

Eugene Koifman commented on HIVE-17660:
---

see if this is still relevant after HIVE-19124

> Compaction for MM runs Cleaner - needs test once IOW is supported
> -
>
> Key: HIVE-17660
> URL: https://issues.apache.org/jira/browse/HIVE-17660
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> Deletion of aborted deltas happens from CompactorMR.run() i.e. from Worker
> but the Worker still sets compaction_queue entry to READY_FOR_CLEANING.
> This is not needed if there are no base_N dirs which can be created by Insert 
> Overwrite
> In this case we can't delete deltas < N until we know no one is reading them, 
> i.e. in Cleaner



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18570) ACID IOW implemented using base may delete too much data

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18570:
-

Assignee: Eugene Koifman

> ACID IOW implemented using base may delete too much data
> 
>
> Key: HIVE-18570
> URL: https://issues.apache.org/jira/browse/HIVE-18570
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Eugene Koifman
>Priority: Blocker
>
> Suppose we have a table with delta_0 insert data.
> Txn 1 starts an insert into delta_1.
> Txn 2 starts an IOW into base_2.
> Txn 2 commits.
> Txn 1 commits after txn 2 but its results would be invisible.
> Txn 2 deletes rows committed by txn 1 that according to standard ACID 
> semantics it could have never observed and affected; this sequence of events 
> is only possible under read-uncommitted isolation level (so, 2 deletes rows 
> written by 1 before 1 commits them). 
> This is if we look at IOW as transactional delete+insert. Otherwise we are 
> just saying IOW performs "semi"-transactional delete.
> If 1 ran an update on rows instead of an insert, and 2 still ran an 
> IOW/delete, row lock conflict (or equivalent) should cause one of them to 
> fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18131) Truncate table for Acid tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458764#comment-16458764
 ] 

Eugene Koifman commented on HIVE-18131:
---

at a minimum this should take an X lock (exclusive write lock when it becomes 
available)

> Truncate table for Acid tables
> --
>
> Key: HIVE-18131
> URL: https://issues.apache.org/jira/browse/HIVE-18131
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> How should this work?  Should it work like Insert Overwrite T select * from T 
> where 1=2?
> This should create a new empty base_x/ and thus operate w/o violating 
> Snapshot Isolation semantics.
> This makes sense for specific partition or unpartitioned table.  What about 
> "Truncate T" where T is partitioned?  Is the expectation to wipe out all 
> partition info or to make each partition empty?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.07.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch, HIVE-19322.06.patch, 
> HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458748#comment-16458748
 ] 

Eugene Koifman commented on HIVE-19322:
---

fix 2 more test outputs in patch 7

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch, HIVE-19322.06.patch, 
> HIVE-19322.07.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.06.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch, HIVE-19322.06.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.05.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457903#comment-16457903
 ] 

Eugene Koifman commented on HIVE-19322:
---

HIVE-19246 modified a bunch of negative golden files.  This patch that fixes 
several TestNegativeMinimrCliDriver tests also restores the old output.  patch 
5 includes reverse of HIVE-19246

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch, HIVE-19322.05.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.04.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch, 
> HIVE-19322.04.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
 Release Note: n/a
   Status: Resolved  (was: Patch Available)

committed to branch-3/master
thanks Sankar for review

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19270.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.03.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Status: Patch Available  (was: Open)

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Status: Open  (was: Patch Available)

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch, HIVE-19322.03.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19258:
--
Component/s: Transactions

> add originals support to MM tables (and make the conversion a metadata only 
> operation)
> --
>
> Key: HIVE-19258
> URL: https://issues.apache.org/jira/browse/HIVE-19258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19258.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17657) export/import for MM tables is broken

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457286#comment-16457286
 ] 

Eugene Koifman commented on HIVE-17657:
---

[~sershe],
left some RB comments - mostly nits, except one about original files

[~sankarh],
MM import will create structure like
{noformat}
├── mm_table_import
│   └── delta_001_001_
│   ├── not_delta_001_001_
│   │   └── 00_0
│   └── not_delta_002_002_
│   └── 00_0
{noformat}
i.e. with subdirs.  does this impact replication at all?


> export/import for MM tables is broken
> -
>
> Key: HIVE-17657
> URL: https://issues.apache.org/jira/browse/HIVE-17657
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-17657.01.patch, HIVE-17657.02.patch, 
> HIVE-17657.03.patch, HIVE-17657.04.patch, HIVE-17657.05.patch, 
> HIVE-17657.patch
>
>
> there is mm_exim.q but it's not clear from the tests what file structure it 
> creates 
> On import the txnids in the directory names would have to be remapped if 
> importing to a different cluster.  Perhaps export can be smart and export 
> highest base_x and accretive deltas (minus aborted ones).  Then import can 
> ...?  It would have to remap txn ids from the archive to new txn ids.  This 
> would then mean that import is made up of several transactions rather than 1 
> atomic op.  (all locks must belong to a transaction)
> One possibility is to open a new txn for each dir in the archive (where 
> start/end txn of file name is the same) and commit all of them at once (need 
> new TMgr API for that).  This assumes using a shared lock (if any!) and thus 
> allows other inserts (not related to import) to occur.
> What if you have delta_6_9, such as a result of concatenate?  If we stipulate 
> that this must mean that there is no delta_6_6 or any other "obsolete" delta 
> in the archive we can map it to a new single txn delta_x_x.
> Add read_only mode for tables (useful in general, may be needed for upgrade 
> etc) and use that to make the above atomic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19279) remove magic directory skipping from CopyTask

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19279:
--
Component/s: Transactions

> remove magic directory skipping from CopyTask
> -
>
> Key: HIVE-19279
> URL: https://issues.apache.org/jira/browse/HIVE-19279
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Follow up from HIVE-17657.
> Code exists in copytask that copies files (fancy that); however, when listing 
> the files, if a single directory exists at the source with no other files, it 
> will skip the directory and copy the files inside instead.
> This directory in various tests is either the "data" directory from export, 
> or some random partition directory ("foo=bar") that if not skipped makes it 
> into the real partition directory at the destination.
> The directory is not skipped if it's not by itself, i.e. any other files or 
> directories are present.
> This seems brittle. Caller of the CopyTask should specify exactly what it 
> wants copied instead of relying on this behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457218#comment-16457218
 ] 

Eugene Koifman commented on HIVE-19322:
---

[~jdere] could you review please

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457211#comment-16457211
 ] 

Eugene Koifman edited comment on HIVE-19322 at 4/27/18 11:31 PM:
-

{noformat}
commit 699c5768c88967abd507122d775bd5955ca45218
Author: Eugene Koifman 
Date:   Tue Apr 17 18:23:13 2018 -0700

HIVE-18739 - Add support for Import/Export from Acid table (Eugene Koifman, 
reviewed by Sergey Shelukhin)

commit bd6b582581de461e620ac9220b862e938dfec8cd
Author: Ashutosh Chauhan 
Date:   Tue Apr 17 18:17:35 2018 -0700

HIVE-19235 : Update golden files for Minimr tests

commit 4cfec3eb9c4c47df6195692ef535b42f9ac36588
Author: Vaibhav Gumashta 
Date:   Tue Apr 17 12:53:40 2018 -0700

HIVE-19126: CachedStore: Use memory estimation to limit cache size during 
prewarm (Vaibhav Gumashta reviewed by Thejas Nair)
{noformat}

If I checkout to just before HIVE-18739 then all the following tests pass and 
fail if I include HIVE-18739. 
{noformat}
mvn test -Dtest=TestNegativeMinimrCliDriver 
-Dqfile=cluster_tasklog_retrieval.q,mapreduce_stack_trace.q,mapreduce_stack_trace_turnoff.q,minimr_broken_pipe.q;
mvn test -Dtest=TestNegativeCliDriver 
-Dqfile=subquery_corr_in_agg.q,subquery_scalar_corr_multi_rows.q,udf_assert_true.q,script_broken_pipe2.q
{noformat}
The attached patch fixes all the tests, however if I checkout HEAD the tests 
TestNegativeMinimrCliDriver are still fixed by the patch but not any of 
TestNegativeCliDriver tests even though the diff that is output is exactly the 
same.  TestNegativeCliDriver clearly could be improved wrt error logging.

So TestNegativeCliDriver will need to be looked at separately in another ticket.


was (Author: ekoifman):
{noformat}
commit 699c5768c88967abd507122d775bd5955ca45218
Author: Eugene Koifman 
Date:   Tue Apr 17 18:23:13 2018 -0700

HIVE-18739 - Add support for Import/Export from Acid table (Eugene Koifman, 
reviewed by Sergey Shelukhin)

commit bd6b582581de461e620ac9220b862e938dfec8cd
Author: Ashutosh Chauhan 
Date:   Tue Apr 17 18:17:35 2018 -0700

HIVE-19235 : Update golden files for Minimr tests

commit 4cfec3eb9c4c47df6195692ef535b42f9ac36588
Author: Vaibhav Gumashta 
Date:   Tue Apr 17 12:53:40 2018 -0700

HIVE-19126: CachedStore: Use memory estimation to limit cache size during 
prewarm (Vaibhav Gumashta reviewed by Thejas Nair)
{noformat}

If I checkout to just before HIVE-18739 then all the following tests pass and 
fail if I include HIVE-18739. 
mvn test -Dtest=TestNegativeMinimrCliDriver 
-Dqfile=cluster_tasklog_retrieval.q,mapreduce_stack_trace.q,mapreduce_stack_trace_turnoff.q,minimr_broken_pipe.q;
mvn test -Dtest=TestNegativeCliDriver 
-Dqfile=subquery_corr_in_agg.q,subquery_scalar_corr_multi_rows.q,udf_assert_true.q,script_broken_pipe2.q

The attached patch fixes all the tests, however if I checkout HEAD the tests 
TestNegativeMinimrCliDriver are still fixed by the patch but not any of 
TestNegativeCliDriver tests even though the diff that is output is exactly the 
same.  TestNegativeCliDriver clearly could be improved wrt error logging.

So TestNegativeCliDriver will need to be looked at separately in another ticket.

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457211#comment-16457211
 ] 

Eugene Koifman commented on HIVE-19322:
---

{noformat}
commit 699c5768c88967abd507122d775bd5955ca45218
Author: Eugene Koifman 
Date:   Tue Apr 17 18:23:13 2018 -0700

HIVE-18739 - Add support for Import/Export from Acid table (Eugene Koifman, 
reviewed by Sergey Shelukhin)

commit bd6b582581de461e620ac9220b862e938dfec8cd
Author: Ashutosh Chauhan 
Date:   Tue Apr 17 18:17:35 2018 -0700

HIVE-19235 : Update golden files for Minimr tests

commit 4cfec3eb9c4c47df6195692ef535b42f9ac36588
Author: Vaibhav Gumashta 
Date:   Tue Apr 17 12:53:40 2018 -0700

HIVE-19126: CachedStore: Use memory estimation to limit cache size during 
prewarm (Vaibhav Gumashta reviewed by Thejas Nair)
{noformat}

If I checkout to just before HIVE-18739 then all the following tests pass and 
fail if I include HIVE-18739. 
mvn test -Dtest=TestNegativeMinimrCliDriver 
-Dqfile=cluster_tasklog_retrieval.q,mapreduce_stack_trace.q,mapreduce_stack_trace_turnoff.q,minimr_broken_pipe.q;
mvn test -Dtest=TestNegativeCliDriver 
-Dqfile=subquery_corr_in_agg.q,subquery_scalar_corr_multi_rows.q,udf_assert_true.q,script_broken_pipe2.q

The attached patch fixes all the tests, however if I checkout HEAD the tests 
TestNegativeMinimrCliDriver are still fixed by the patch but not any of 
TestNegativeCliDriver tests even though the diff that is output is exactly the 
same.  TestNegativeCliDriver clearly could be improved wrt error logging.

So TestNegativeCliDriver will need to be looked at separately in another ticket.

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.02.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Status: Patch Available  (was: Open)

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.02.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: (was: HIVE-19322.01.patch)

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Attachment: HIVE-19322.01.patch

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19322.01.patch
>
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Status: Patch Available  (was: Open)

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19270.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456610#comment-16456610
 ] 

Eugene Koifman commented on HIVE-19270:
---

[~sankarh], correct one attached

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19270.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Attachment: HIVE-19270.01.patch

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19270.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Attachment: (was: HIVE-19315.01.patch)

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Status: Open  (was: Patch Available)

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455227#comment-16455227
 ] 

Eugene Koifman commented on HIVE-19322:
---

the same change doesn't fix these tests so it's not caused by  HIVE-18739.

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454854#comment-16454854
 ] 

Eugene Koifman commented on HIVE-19322:
---

[~vgarg], I checked the tests you posted - not 1 of them has the same error...

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Affects Version/s: 3.0.0
   Status: Patch Available  (was: Open)

this is a test only change.
Something changed in compiler/runtime that writes rows in different orders so 
they get row_ids assigned in a different order.  Change is not significant.

[~sankarh] could you review please

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19315.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19270:
--
Attachment: HIVE-19315.01.patch

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19315.01.patch
>
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454804#comment-16454804
 ] 

Eugene Koifman commented on HIVE-19124:
---

+1

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.08.patch, HIVE-19124.09.patch, HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19270) TestAcidOnTez tests are failing

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-19270:
-

Assignee: Eugene Koifman

> TestAcidOnTez tests are failing
> ---
>
> Key: HIVE-19270
> URL: https://issues.apache.org/jira/browse/HIVE-19270
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vineet Garg
>Assignee: Eugene Koifman
>Priority: Major
>
> Following tests are failing:
> * testCtasTezUnion
> * testNonStandardConversion01
> * testAcidInsertWithRemoveUnion
> All of them have the similar failure:
> {noformat}
> Actual line 0 ac: {"writeid":1,"bucketid":536870913,"rowid":1} 1 2 
> file:/home/hiveptest/35.193.47.6-hiveptest-1/apache-github-source-source/itests/hive-unit/target/tmp/org.apache.hadoop.hive.ql.TestAcidOnTez-1524409020904/warehouse/t/delta_001_001_0001/bucket_0
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Description: 
this is apparently caused by HIVE-18739, specifically changing

{{private static ThreadLocal tss}} in {{SessionState}} to 
{{private static InheritableThreadLocal tss}}

need to figure out why this is.  

Looks like
{{TestNegativeMinimrCliDriver 
-Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
are also broken by this


  was:
this is apparently caused by HIVE-18739, specifically changing

{{private static ThreadLocal tss}} in {{SessionState}} to 
{{private static InheritableThreadLocal tss}}

need to figure out why this is.  

Looks like
TestNegativeMinimrCliDriver 
-Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q
are also broken by this



> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> {{TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q}}
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19322:
--
Description: 
this is apparently caused by HIVE-18739, specifically changing

{{private static ThreadLocal tss}} in {{SessionState}} to 
{{private static InheritableThreadLocal tss}}

need to figure out why this is.  

Looks like
TestNegativeMinimrCliDriver 
-Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q
are also broken by this


  was:
this is apparently caused by HIVE-18739, specifically changing

{{private static ThreadLocal tss}} in {{SessionState}} to 
{{private static InheritableThreadLocal tss}}

need to figure out why this is.  



> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  
> Looks like
> TestNegativeMinimrCliDriver 
> -Dqfile=mapreduce_stack_trace_turnoff.q,mapreduce_stack_trace.q,cluster_tasklog_retrieval.q
> are also broken by this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18739) Add support for Import/Export from Acid table

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454679#comment-16454679
 ] 

Eugene Koifman commented on HIVE-18739:
---

filed HIVE-19322

> Add support for Import/Export from Acid table
> -
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-18739.01-branch-3.patch, HIVE-18739.01.patch, 
> HIVE-18739.02-branch-3.patch, HIVE-18739.04.patch, HIVE-18739.06.patch, 
> HIVE-18739.08.patch, HIVE-18739.09.patch, HIVE-18739.10.patch, 
> HIVE-18739.11.patch, HIVE-18739.12.patch, HIVE-18739.13.patch, 
> HIVE-18739.14.patch, HIVE-18739.15.patch, HIVE-18739.16.patch, 
> HIVE-18739.17.patch, HIVE-18739.19.patch, HIVE-18739.20.patch, 
> HIVE-18739.21.patch, HIVE-18739.23.patch, HIVE-18739.24.patch, 
> HIVE-18739.25.patch, HIVE-18739.26.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19322) broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-19322:
-

Assignee: Eugene Koifman

> broken test: TestNegativeMinimrCliDriver#testCliDriver[minimr_broken_pipe]
> --
>
> Key: HIVE-19322
> URL: https://issues.apache.org/jira/browse/HIVE-19322
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> this is apparently caused by HIVE-18739, specifically changing
> {{private static ThreadLocal tss}} in {{SessionState}} to 
> {{private static InheritableThreadLocal tss}}
> need to figure out why this is.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HIVE-19286) NPE in MERGE operator on MR mode

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454659#comment-16454659
 ] 

Eugene Koifman edited comment on HIVE-19286 at 4/26/18 6:16 PM:


it's odd that something is trying to call 
UDFToString on struct (i.e. the 
ROW__ID)
The update/delete branches of the multi-insert should be caling UDFToInteger 
which does have special handling of ROW__ID


was (Author: ekoifman):
it's odd that it's trying to call 
UDFToString on struct (i.e. the 
ROW__ID)
if anything it should be be UDFToInteger which does have special handling of 
ROW__ID

> NPE in MERGE operator on MR mode
> 
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer, Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai 
> Date:   Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING, 
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES 
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002', 
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age  FROM customer_target;
> {code}
> {code}
> +--+-++--+
> |  id  | first_name  | last_name  | age  |
> +--+-++--+
> | 002  | Michael | Watson | 27   |
> | 001  | John| Smith  | 45   |
> | 003  | Den | Brown  | 33   |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING, 
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007', 
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age  FROM customer_source;
> {code}
> {code}
> +--+-++--+
> |  id  | first_name  | last_name  | age  |
> +--+-++--+
> | 088  | Robert  | Dowson | 25   |
> | 001  | Dorothi | Hogward| 77   |
> | 007  | Alex| Bowee  | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id 
> when matched then update set first_name = src.first_name, last_name = 
> src.last_name when not matched then insert values (src.id, src.first_name, 
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:  method=deserializePlan 
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448  INFO [main] exec.SerializationUtilities: 
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or 
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:  method=deserializePlan start=1524568304448 end=1524568304538 duration=90 
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545  INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44  
>   Starting to launch local task to process map join;  maximum memory = 
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators: 
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or 
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676  INFO [main] mr.MapredLocalTask: fetchoperator for 
> trg created
> 2018-04-24T07:11:44,676  INFO [main] exec.TableScanOperator: Initializing 
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization 
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS 
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing 
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing 
> child 37 HASHTABLESINK
> 

[jira] [Commented] (HIVE-19286) NPE in MERGE operator on MR mode

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454659#comment-16454659
 ] 

Eugene Koifman commented on HIVE-19286:
---

it's odd that it's trying to call 
UDFToString on struct (i.e. the 
ROW__ID)
if anything it should be be UDFToInteger which does have special handling of 
ROW__ID

> NPE in MERGE operator on MR mode
> 
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer, Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai 
> Date:   Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING, 
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES 
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002', 
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age  FROM customer_target;
> {code}
> {code}
> +--+-++--+
> |  id  | first_name  | last_name  | age  |
> +--+-++--+
> | 002  | Michael | Watson | 27   |
> | 001  | John| Smith  | 45   |
> | 003  | Den | Brown  | 33   |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING, 
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007', 
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age  FROM customer_source;
> {code}
> {code}
> +--+-++--+
> |  id  | first_name  | last_name  | age  |
> +--+-++--+
> | 088  | Robert  | Dowson | 25   |
> | 001  | Dorothi | Hogward| 77   |
> | 007  | Alex| Bowee  | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id 
> when matched then update set first_name = src.first_name, last_name = 
> src.last_name when not matched then insert values (src.id, src.first_name, 
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:  method=deserializePlan 
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448  INFO [main] exec.SerializationUtilities: 
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or 
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:  method=deserializePlan start=1524568304448 end=1524568304538 duration=90 
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545  INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44  
>   Starting to launch local task to process map join;  maximum memory = 
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators: 
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or 
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676  INFO [main] mr.MapredLocalTask: fetchoperator for 
> trg created
> 2018-04-24T07:11:44,676  INFO [main] exec.TableScanOperator: Initializing 
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization 
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS 
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing 
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing 
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676  INFO [main] exec.HashTableSinkOperator: Initializing 
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677  INFO [main] mapjoin.MapJoinMemoryExhaustionHandler: 
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error: 
> Map local work failed
> 

[jira] [Resolved] (HIVE-19315) Test failure org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-19315.
---
   Resolution: Fixed
Fix Version/s: 3.0.0
 Release Note: n/a

committed to branch-3/master
thanks Jason for the review

> Test failure 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3
> --
>
> Key: HIVE-19315
> URL: https://issues.apache.org/jira/browse/HIVE-19315
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Tests, Transactions
>Affects Versions: 3.1.0
>Reporter: Pravin Dsilva
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19315.01.patch
>
>
> *Error Message*
> {code:java}
> Exception msg didn't match expected:<...conflict on default/[TAB_PART]/p=blah 
> committed by...> but was:<...conflict on default/[tab_part]/p=blah committed 
> by...>
> {code}
> *Stacktrace*
> {code:java}
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 22.704 s <<< FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] 
> testWriteSetTracking3(org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2)  
> Time elapsed: 19.501 s  <<< FAILURE!
> org.junit.ComparisonFailure: Exception msg didn't match expected:<...conflict 
> on default/[TAB_PART]/p=blah committed by...> but was:<...conflict on 
> default/[tab_part]/p=blah committed by...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3(TestDbTxnManager2.java:994)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19315) Test failure org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19315:
--
Component/s: Transactions

> Test failure 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3
> --
>
> Key: HIVE-19315
> URL: https://issues.apache.org/jira/browse/HIVE-19315
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Tests, Transactions
>Affects Versions: 3.1.0
>Reporter: Pravin Dsilva
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19315.01.patch
>
>
> *Error Message*
> {code:java}
> Exception msg didn't match expected:<...conflict on default/[TAB_PART]/p=blah 
> committed by...> but was:<...conflict on default/[tab_part]/p=blah committed 
> by...>
> {code}
> *Stacktrace*
> {code:java}
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 22.704 s <<< FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] 
> testWriteSetTracking3(org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2)  
> Time elapsed: 19.501 s  <<< FAILURE!
> org.junit.ComparisonFailure: Exception msg didn't match expected:<...conflict 
> on default/[TAB_PART]/p=blah committed by...> but was:<...conflict on 
> default/[tab_part]/p=blah committed by...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3(TestDbTxnManager2.java:994)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19315) Test failure org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454546#comment-16454546
 ] 

Eugene Koifman commented on HIVE-19315:
---

issue caused by HIVE-19260
trivial test only change

> Test failure 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3
> --
>
> Key: HIVE-19315
> URL: https://issues.apache.org/jira/browse/HIVE-19315
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Tests
>Affects Versions: 3.1.0
>Reporter: Pravin Dsilva
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19315.01.patch
>
>
> *Error Message*
> {code:java}
> Exception msg didn't match expected:<...conflict on default/[TAB_PART]/p=blah 
> committed by...> but was:<...conflict on default/[tab_part]/p=blah committed 
> by...>
> {code}
> *Stacktrace*
> {code:java}
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 22.704 s <<< FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] 
> testWriteSetTracking3(org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2)  
> Time elapsed: 19.501 s  <<< FAILURE!
> org.junit.ComparisonFailure: Exception msg didn't match expected:<...conflict 
> on default/[TAB_PART]/p=blah committed by...> but was:<...conflict on 
> default/[tab_part]/p=blah committed by...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3(TestDbTxnManager2.java:994)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19315) Test failure org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19315:
--
Attachment: HIVE-19315.01.patch

> Test failure 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3
> --
>
> Key: HIVE-19315
> URL: https://issues.apache.org/jira/browse/HIVE-19315
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Tests
>Affects Versions: 3.1.0
>Reporter: Pravin Dsilva
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19315.01.patch
>
>
> *Error Message*
> {code:java}
> Exception msg didn't match expected:<...conflict on default/[TAB_PART]/p=blah 
> committed by...> but was:<...conflict on default/[tab_part]/p=blah committed 
> by...>
> {code}
> *Stacktrace*
> {code:java}
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 22.704 s <<< FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] 
> testWriteSetTracking3(org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2)  
> Time elapsed: 19.501 s  <<< FAILURE!
> org.junit.ComparisonFailure: Exception msg didn't match expected:<...conflict 
> on default/[TAB_PART]/p=blah committed by...> but was:<...conflict on 
> default/[tab_part]/p=blah committed by...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3(TestDbTxnManager2.java:994)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-19315) Test failure org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-19315:
-

Assignee: Eugene Koifman

> Test failure 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2#testWriteSetTracking3
> --
>
> Key: HIVE-19315
> URL: https://issues.apache.org/jira/browse/HIVE-19315
> Project: Hive
>  Issue Type: Bug
>  Components: Test, Tests
>Affects Versions: 3.1.0
>Reporter: Pravin Dsilva
>Assignee: Eugene Koifman
>Priority: Major
>
> *Error Message*
> {code:java}
> Exception msg didn't match expected:<...conflict on default/[TAB_PART]/p=blah 
> committed by...> but was:<...conflict on default/[tab_part]/p=blah committed 
> by...>
> {code}
> *Stacktrace*
> {code:java}
> [INFO] ---
> [INFO]  T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 22.704 s <<< FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2
> [ERROR] 
> testWriteSetTracking3(org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2)  
> Time elapsed: 19.501 s  <<< FAILURE!
> org.junit.ComparisonFailure: Exception msg didn't match expected:<...conflict 
> on default/[TAB_PART]/p=blah committed by...> but was:<...conflict on 
> default/[tab_part]/p=blah committed by...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at 
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3(TestDbTxnManager2.java:994)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19312:
--
Component/s: Transactions

> MM tables don't work with BucketizedHIF
> ---
>
> Key: HIVE-19312
> URL: https://issues.apache.org/jira/browse/HIVE-19312
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19312.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19211) New streaming ingest API and support for dynamic partitioning

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453363#comment-16453363
 ] 

Eugene Koifman commented on HIVE-19211:
---

left RB comments - a lot of nits but not all

> New streaming ingest API and support for dynamic partitioning
> -
>
> Key: HIVE-19211
> URL: https://issues.apache.org/jira/browse/HIVE-19211
> Project: Hive
>  Issue Type: Sub-task
>  Components: Streaming
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, 
> HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, 
> HIVE-19211.6.patch, HIVE-19211.7.patch
>
>
> - New streaming API under new hive sub-module
> - Dynamic partitioning support
> - Auto-rollover transactions
> - Automatic heartbeating



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18193:
--
Fix Version/s: (was: 3.1.0)

> Migrate existing ACID tables to use write id per table rather than global 
> transaction id
> 
>
> Key: HIVE-18193
> URL: https://issues.apache.org/jira/browse/HIVE-18193
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Blocker
>  Labels: ACID, Upgrade
>
> dependent upon HIVE-18192
> For existing ACID Tables we need to update the table level write id 
> metatables/sequences so any new operations on these tables works seamlessly 
> without any conflicting data in existing base/delta files.
> 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID.
> 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is 
> set to current value of NEXT_TXN_ID.NTXN_NEXT.
> 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID 
> such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId.
> 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in 
> COMPLETED_TXN_COMPONENTS to store the write id which should be set as 
> respective values of TC_TXNID and CTC_TXNID from the same row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452979#comment-16452979
 ] 

Eugene Koifman commented on HIVE-18193:
---

this is blocker for 3.0

> Migrate existing ACID tables to use write id per table rather than global 
> transaction id
> 
>
> Key: HIVE-18193
> URL: https://issues.apache.org/jira/browse/HIVE-18193
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Blocker
>  Labels: ACID, Upgrade
>
> dependent upon HIVE-18192
> For existing ACID Tables we need to update the table level write id 
> metatables/sequences so any new operations on these tables works seamlessly 
> without any conflicting data in existing base/delta files.
> 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID.
> 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is 
> set to current value of NEXT_TXN_ID.NTXN_NEXT.
> 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID 
> such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId.
> 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in 
> COMPLETED_TXN_COMPONENTS to store the write id which should be set as 
> respective values of TC_TXNID and CTC_TXNID from the same row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18193:
--
Priority: Blocker  (was: Major)

> Migrate existing ACID tables to use write id per table rather than global 
> transaction id
> 
>
> Key: HIVE-18193
> URL: https://issues.apache.org/jira/browse/HIVE-18193
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: Sankar Hariappan
>Priority: Blocker
>  Labels: ACID, Upgrade
>
> dependent upon HIVE-18192
> For existing ACID Tables we need to update the table level write id 
> metatables/sequences so any new operations on these tables works seamlessly 
> without any conflicting data in existing base/delta files.
> 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID.
> 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is 
> set to current value of NEXT_TXN_ID.NTXN_NEXT.
> 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID 
> such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId.
> 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in 
> COMPLETED_TXN_COMPONENTS to store the write id which should be set as 
> respective values of TC_TXNID and CTC_TXNID from the same row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452978#comment-16452978
 ] 

Eugene Koifman commented on HIVE-19260:
---

committed to branch-3

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19260:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452891#comment-16452891
 ] 

Eugene Koifman commented on HIVE-19124:
---

LGTM.  Thank you.

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.08.patch, HIVE-19124.09.patch, HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19233:
--
Attachment: HIVE-19233.04.patch

> Add utility for acid 1.0 to 2.0 migration
> -
>
> Key: HIVE-19233
> URL: https://issues.apache.org/jira/browse/HIVE-19233
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19233.01.patch, HIVE-19233.02.patch, 
> HIVE-19233.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19233:
--
Status: Patch Available  (was: Open)

> Add utility for acid 1.0 to 2.0 migration
> -
>
> Key: HIVE-19233
> URL: https://issues.apache.org/jira/browse/HIVE-19233
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19233.01.patch, HIVE-19233.02.patch, 
> HIVE-19233.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451283#comment-16451283
 ] 

Eugene Koifman commented on HIVE-19124:
---

My point was that instead of letting the Driver create the list, the list 
created by Worker should be passed to the Driver.  That way it doesn't have to 
extract any ids, does it?  (and the flow between MM and acid remains more 
similar)
This is the code from Worker that I'm referring to 
{noformat}
String fullTableName = TxnUtils.getFullTableName(t.getDbName(), 
t.getTableName());
GetValidWriteIdsRequest rqst = new 
GetValidWriteIdsRequest(Collections.singletonList(fullTableName), null);
final ValidWriteIdList tblValidWriteIds =

TxnUtils.createValidCompactWriteIdList(txnHandler.getValidWriteIds(rqst).getTblValidWriteIds().get(0));
LOG.debug("ValidCompactWriteIdList: " + 
tblValidWriteIds.writeToString());
txnHandler.setCompactionHighestWriteId(ci, 
tblValidWriteIds.getHighWatermark());
{noformat}


> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.08.patch, HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451231#comment-16451231
 ] 

Eugene Koifman commented on HIVE-19124:
---

a simpler approach would have been to address HIVE-18570 first by requiring X 
lock IOW, then by definition there would be no open txns at the time IOW runs 
(and similarly no open write txns when X lock is replaced by Write lock which 
still allows readers).  Unfortunately it's no longer possible since HIVE-18825 
moved snapshot acquisition to be before lock acquisition.

I think a larger issue (including Gopal's proposal) is that it gets the wrong 
HWM.  If you look at Worker,
it creates a Compaction writeIldList and stores the HWM in the metastore.  This 
is important so that Cleaner later doesn't clean above this mark, i.e. the 
highest level compaction looked at.

In this patch, the MM compactor IOW query creates it's own Snapshot which may 
be different.  Perhaps a more consistent approach is to set the snapshot on the 
Driver in compaction mode from Worker to make this consistent. 

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450383#comment-16450383
 ] 

Eugene Koifman commented on HIVE-19124:
---

I saw the comment.  That isn't acceptable.  

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450373#comment-16450373
 ] 

Eugene Koifman commented on HIVE-19124:
---

Hold on.  We already have a mechanism for compactor to make sure it doesn't 
compact above an open txn:
TxnUtils.createValidCompactWriteIdList() .
This reinvents the wheel and add a new one to do the same thing in the Reader 
list.  Why is this good?

> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, 
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, 
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, 
> HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be 
> supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19137) orcfiledump doesn't print hive.acid.version value

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19137:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
 Release Note: n/a
   Status: Resolved  (was: Patch Available)

committed to branch-3/master
thanks Igor for the contribution

> orcfiledump doesn't print hive.acid.version value
> -
>
> Key: HIVE-19137
> URL: https://issues.apache.org/jira/browse/HIVE-19137
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19137-branch-3.01.patch, HIVE-19137.01.patch, 
> HIVE-19137.02-branch-3.patch, HIVE-19137.02.patch, 
> HIVE-19137.03-branch-3.patch, HIVE-19137.03.patch, HIVE-19137.04.patch, 
> HIVE-19137.05.patch
>
>
> HIVE-18659 added hive.acid.version in the file footer.  
> orcfiledump prints something like 
> {noformat}
> User Metadata:
>   hive.acid.key.index=1,536870912,1;
>   hive.acid.stats=2,0,0
>   hive.acid.version=
> {noformat}
> probably because
> {noformat}
> public static void setAcidVersionInDataFile(Writer writer) {
>   //so that we know which version wrote the file
>   ByteBuffer bf = ByteBuffer.allocate(4).putInt(ORC_ACID_VERSION);
>   bf.rewind(); //don't ask - some ByteBuffer weridness. w/o this, empty 
> buffer is written
>   writer.addUserMetadata(ACID_VERSION_KEY, bf);
> }
> {noformat}
> use 
> {{UTF8.encode())}} instead



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450362#comment-16450362
 ] 

Eugene Koifman commented on HIVE-19260:
---

Thanks Prasanth for the review
committed to master

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19137) orcfiledump doesn't print hive.acid.version value

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450339#comment-16450339
 ] 

Eugene Koifman commented on HIVE-19137:
---

+1

> orcfiledump doesn't print hive.acid.version value
> -
>
> Key: HIVE-19137
> URL: https://issues.apache.org/jira/browse/HIVE-19137
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19137-branch-3.01.patch, HIVE-19137.01.patch, 
> HIVE-19137.02-branch-3.patch, HIVE-19137.02.patch, 
> HIVE-19137.03-branch-3.patch, HIVE-19137.03.patch, HIVE-19137.04.patch, 
> HIVE-19137.05.patch
>
>
> HIVE-18659 added hive.acid.version in the file footer.  
> orcfiledump prints something like 
> {noformat}
> User Metadata:
>   hive.acid.key.index=1,536870912,1;
>   hive.acid.stats=2,0,0
>   hive.acid.version=
> {noformat}
> probably because
> {noformat}
> public static void setAcidVersionInDataFile(Writer writer) {
>   //so that we know which version wrote the file
>   ByteBuffer bf = ByteBuffer.allocate(4).putInt(ORC_ACID_VERSION);
>   bf.rewind(); //don't ask - some ByteBuffer weridness. w/o this, empty 
> buffer is written
>   writer.addUserMetadata(ACID_VERSION_KEY, bf);
> }
> {noformat}
> use 
> {{UTF8.encode())}} instead



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19260:
--
Attachment: HIVE-19260.01-branch-3.patch

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450170#comment-16450170
 ] 

Eugene Koifman commented on HIVE-19260:
---

yes, it guaranteed to be not null

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19286:
--
Component/s: Transactions

> NPE in MERGE operator on MR mode
> 
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai 
> Date:   Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING, 
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES 
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002', 
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age  FROM customer_target;
> {code}
> {code}
> +--+-++--+
> |  id  | first_name  | last_name  | age  |
> +--+-++--+
> | 002  | Michael | Watson | 27   |
> | 001  | John| Smith  | 45   |
> | 003  | Den | Brown  | 33   |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING, 
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007', 
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age  FROM customer_source;
> {code}
> {code}
> +--+-++--+
> |  id  | first_name  | last_name  | age  |
> +--+-++--+
> | 088  | Robert  | Dowson | 25   |
> | 001  | Dorothi | Hogward| 77   |
> | 007  | Alex| Bowee  | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id 
> when matched then update set first_name = src.first_name, last_name = 
> src.last_name when not matched then insert values (src.id, src.first_name, 
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:  method=deserializePlan 
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448  INFO [main] exec.SerializationUtilities: 
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or 
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:  method=deserializePlan start=1524568304448 end=1524568304538 duration=90 
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545  INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44  
>   Starting to launch local task to process map join;  maximum memory = 
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators: 
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or 
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676  INFO [main] mr.MapredLocalTask: fetchoperator for 
> trg created
> 2018-04-24T07:11:44,676  INFO [main] exec.TableScanOperator: Initializing 
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization 
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS 
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing 
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing 
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676  INFO [main] exec.HashTableSinkOperator: Initializing 
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677  INFO [main] mapjoin.MapJoinMemoryExhaustionHandler: 
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error: 
> Map local work failed
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
>  ~[hive-exec-2.3.3.jar:2.3.3]
> at 
> 

[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446570#comment-16446570
 ] 

Eugene Koifman commented on HIVE-19260:
---

partition column names are never specified by caller - they are taken from 
table metdata which is normalize and I don't think lower casing partition 
values is a good idea.  I don't see any LockComparator changes here.
in fact, findRelatedLocks() is not even necessary given the other changes - 
being a bit paranoid here.

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446560#comment-16446560
 ] 

Eugene Koifman commented on HIVE-19260:
---

[~prasanth_j] could you review please

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-19260:
--
Status: Patch Available  (was: Open)

> Streaming Ingest API doesn't normalize db.table names
> -
>
> Key: HIVE-19260
> URL: https://issues.apache.org/jira/browse/HIVE-19260
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Streaming, Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-19260.01.patch
>
>
> Hive is case insensitive wrt db/table names.  These gets normalized to lower 
> case for SQL processing.
> When HiveEndPoint is created it uses db.table strings as is, and they end up 
> propagated this way to transaction metadata tables in the metastore via lock 
> acquisition.  This makes them look like different tables in Cleaner and lock 
> manager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)