MaxGekk commented on pull request #30643: URL: https://github.com/apache/spark/pull/30643#issuecomment-743908391
I guess there is bug in HiveMetaStore.java. The method `drop_partition_common()` tries to remove parent folders recursively. On my laptop, I prohibited to remove anything above of warehouse, and the method fails because it tries to remove parent folders out of the warehouse folder: <img width="1111" alt="Screenshot 2020-12-13 at 00 28 59" src="https://user-images.githubusercontent.com/1580697/101995322-384c6580-3cda-11eb-85c1-b579b4ed229b.png";> where: ``` partPath.getParent() = /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/warehouse-531bebac-eb8e-4690-af89-50c001e6c684/ns.db/tbl part_vals.size() = 10 ``` The method: ``` private void deleteParentRecursive(Path parent, int depth, boolean mustPurge) throws IOException, MetaException { if (depth > 0 && parent != null && wh.isWritable(parent)) { if (wh.isDir(parent) && wh.isEmpty(parent)) { wh.deleteDir(parent, true, mustPurge); } deleteParentRecursive(parent.getParent(), depth - 1, mustPurge); } } ``` goes up and up, and tries to remove: ``` /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/warehouse-531bebac-eb8e-4690-af89-50c001e6c684/ns.db /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/warehouse-531bebac-eb8e-4690-af89-50c001e6c684 /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn /private/var/folders/p3 ``` On my laptop, it fails at some point on the call `wh.isEmpty(parent)` since I prohibited access to some folders from IntelliJ IDEA, but it is not clear what happens on Jenkins machines. The method `deleteParentRecursive` could work if it would get the correct path. ... though maybe the test MUST set the correct location which ALL partition parts there https://github.com/apache/spark/blob/fab2995972761503563fa2aa547c67047c51bd33/sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableAddPartitionSuiteBase.scala#L157 . Let me review all tests where we set the location like `LOCATION 'loc1'`, probably that's wrong. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/1580697/101995322-384c6580-3cda-11eb-85c1-b579b4ed229b.png"](https://user-images.githubusercontent.com/1580697/101995322-384c6580-3cda-11eb-85c1-b579b4ed229b.png"){kind=link}