[jira] [Commented] (HIVE-20460) AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables

2018-10-31 Thread Eugene Koifman (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670775#comment-16670775
 ] 

Eugene Koifman commented on HIVE-20460:
---

the system isn't quite ready for this.  full CRUD still relies on rename().

> AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables
> --
>
> Key: HIVE-20460
> URL: https://issues.apache.org/jira/browse/HIVE-20460
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{Directory.getAbortedDirectories()}} lists deltas where all txns in the 
> range are aborted.
> These are then purged by {{Worker}} (\{{CompactorMR}} but only for 
> insert-only tables.
> Full CRUD tables currently rely on {{FileSystem.rename()}} in {{MoveTask}} 
> and so no reader (or {{Cleaner}} should every see a delta where all data is 
> aborted.  
>  
> Once rename() is eliminated for full CRUD (just like insert-only) 
> transactional tables, Cleaner (or Worker) should take care of these.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20460) AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables

2018-10-31 Thread Igor Kryvenko (JIRA) 


[ 
https://issues.apache.org/jira/browse/HIVE-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670648#comment-16670648
 ] 

Igor Kryvenko commented on HIVE-20460:
--

Hi [~ekoifman] . Can I take this JIRA?

> AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables
> --
>
> Key: HIVE-20460
> URL: https://issues.apache.org/jira/browse/HIVE-20460
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{Directory.getAbortedDirectories()}} lists deltas where all txns in the 
> range are aborted.
> These are then purged by {{Worker}} (\{{CompactorMR}} but only for 
> insert-only tables.
> Full CRUD tables currently rely on {{FileSystem.rename()}} in {{MoveTask}} 
> and so no reader (or {{Cleaner}} should every see a delta where all data is 
> aborted.  
>  
> Once rename() is eliminated for full CRUD (just like insert-only) 
> transactional tables, Cleaner (or Worker) should take care of these.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)