[ 
https://issues.apache.org/jira/browse/HIVE-18696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428922#comment-16428922
 ] 

Hive QA commented on HIVE-18696:
--------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
25s{color} | {color:red} standalone-metastore: The patch generated 2 new + 408 
unchanged - 3 fixed = 410 total (was 411) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
15s{color} | {color:red} The patch generated 50 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 47s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10042/dev-support/hive-personality.sh
 |
| git revision | master / 4931805 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10042/yetus/diff-checkstyle-standalone-metastore.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10042/yetus/patch-asflicense-problems.txt
 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-10042/yetus.txt |
| Powered by | Apache Yetus    http://yetus.apache.org |


This message was automatically generated.



> The partition folders might not get cleaned up properly in the 
> HiveMetaStore.add_partitions_core method if an exception occurs
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-18696
>                 URL: https://issues.apache.org/jira/browse/HIVE-18696
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>            Reporter: Marta Kuczora
>            Assignee: Marta Kuczora
>            Priority: Major
>         Attachments: HIVE-18696.1.patch, HIVE-18696.2.patch, 
> HIVE-18696.3.patch, HIVE-18696.4.patch, HIVE-18696.5.patch, HIVE-18696.6.patch
>
>
> When trying to add multiple partitions, but one of them cannot be created 
> successfully, none of the partitions are created, but the folders might not 
> be cleaned up properly. See the test case "testAddPartitionsOneInvalid" in 
> the TestAddPartitions test.
> This is the problematic code in the HiveMetaStore.add_partitions_core method:
> {code:java}
>         for (final Partition part : parts) {
>           if (!part.getTableName().equals(tblName) || 
> !part.getDbName().equals(dbName)) {
>             throw new MetaException("Partition does not belong to target 
> table "
>                 + dbName + "." + tblName + ": " + part);
>           }
>           boolean shouldAdd = startAddPartition(ms, part, ifNotExists);
>           if (!shouldAdd) {
>             existingParts.add(part);
>             LOG.info("Not adding partition " + part + " as it already 
> exists");
>             continue;
>           }
>           final UserGroupInformation ugi;
>           try {
>             ugi = UserGroupInformation.getCurrentUser();
>           } catch (IOException e) {
>             throw new RuntimeException(e);
>           }
>           partFutures.add(threadPool.submit(new Callable<Partition>() {
>             @Override
>             public Partition call() throws Exception {
>               ugi.doAs(new PrivilegedExceptionAction<Object>() {
>                 @Override
>                 public Object run() throws Exception {
>                   try {
>                     boolean madeDir = createLocationForAddedPartition(table, 
> part);
>                     if (addedPartitions.put(new PartValEqWrapper(part), 
> madeDir) != null) {
>                       // Technically, for ifNotExists case, we could insert 
> one and discard the other
>                       // because the first one now "exists", but it seems 
> better to report the problem
>                       // upstream as such a command doesn't make sense.
>                       throw new MetaException("Duplicate partitions in the 
> list: " + part);
>                     }
>                     initializeAddedPartition(table, part, madeDir);
>                   } catch (MetaException e) {
>                     throw new IOException(e.getMessage(), e);
>                   }
>                   return null;
>                 }
>               });
>               return part;
>             }
>           }));
>         }
> {code}
> When going through the partitions, let's say for the first two partitions the 
> threads are successfully submitted to create the folders. But an exception 
> occurs for the third partition in the code before submitting the thread. (It 
> can happen if the partition has different table or db name as the others or 
> it has invalid value.)
>  In this case the execution will jump to the finally part where the folders 
> in the "addedPartitions" map will be cleaned up. However it can happen that 
> the threads for the first two partitions are not finished with the folder 
> creation yet, so the map can be empty or it can contain only one of the 
> partitions.
> This issue also happens in the HiveMetastore.add_partitions_pspec_core 
> method, as this code part is the same as in the add_partitions_core method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to