[
https://issues.apache.org/jira/browse/HIVE-21052?focusedWorklogId=495202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495202
]
ASF GitHub Bot logged work on HIVE-21052:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Oct/20 07:53
Start Date: 05/Oct/20 07:53
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1548:
URL: https://github.com/apache/hive/pull/1548#discussion_r499404466
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##########
@@ -2839,6 +2848,87 @@ public static void setNonTransactional(Map<String,
String> tblProps) {
tblProps.remove(hive_metastoreConstants.TABLE_TRANSACTIONAL_PROPERTIES);
}
+ /**
+ * Look for delta directories matching the list of writeIds and deletes them.
+ * @param rootPartition root partition to look for the delta directories
+ * @param conf configuration
+ * @param writeIds list of writeIds to look for in the delta directories
+ * @return list of deleted directories.
+ * @throws IOException
+ */
+ public static List<FileStatus> deleteDeltaDirectories(Path rootPartition,
Configuration conf, Set<Long> writeIds)
+ throws IOException {
+ FileSystem fs = rootPartition.getFileSystem(conf);
+
+ PathFilter filter = (p) -> {
+ String name = p.getName();
+ for (Long wId : writeIds) {
+ if (name.startsWith(deltaSubdir(wId, wId)) && !name.contains("=")) {
+ return true;
+ } else if (name.startsWith(baseDir(wId)) && !name.contains("=")) {
+ return true;
+ }
+ }
+ return false;
+ };
+ List<FileStatus> deleted = new ArrayList<>();
+ deleteDeltaDirectoriesAux(rootPartition, fs, filter, deleted);
+ return deleted;
+ }
+
+ private static void deleteDeltaDirectoriesAux(Path root, FileSystem fs,
PathFilter filter, List<FileStatus> deleted)
+ throws IOException {
+ RemoteIterator<FileStatus> it = listIterator(fs, root, null);
+
+ while (it.hasNext()) {
+ FileStatus fStatus = it.next();
+ if (fStatus.isDirectory()) {
+ if (filter.accept(fStatus.getPath())) {
+ fs.delete(fStatus.getPath(), true);
+ deleted.add(fStatus);
+ } else {
+ deleteDeltaDirectoriesAux(fStatus.getPath(), fs, filter, deleted);
+ if (isDirectoryEmpty(fs, fStatus.getPath())) {
Review comment:
Are we doing this to delete newly created partitions if there are no
other writes? Is this ok, what if we found a valid empty partition that is
registered in the HMS? We should not delete that. I think this can be skipped
all together, the empty partition dir will not bother anybody
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 495202)
Time Spent: 1h 50m (was: 1h 40m)
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -------------------------------------------------------------------------------------
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 3.0.0, 3.1.1
> Reporter: Jaume M
> Assignee: Jaume M
> Priority: Critical
> Labels: pull-request-available
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch,
> HIVE-21052.8.patch, HIVE-21052.9.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
