[
https://issues.apache.org/jira/browse/FALCON-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912511#comment-13912511
]
Samarth Gupta edited comment on FALCON-321 at 2/26/14 4:46 AM:
---------------------------------------------------------------
[~raghavgautam] [~svenkat] this is intentional and a feature currently. We
dont want to leave too many empty folder on HDFS. This was not happening
earlier and have been enabled now.
In fact at inmobi we added in regression to ensure that.
Feature is open for debate ..... :)
was (Author: samarthg):
this is intentional and a feature currently. We dont want to leave too many
empty folder on HDFS. This was not happening earlier and have been enabled
now.
Feature is open for debate ..... :)
> Feed evictor deleting more stuff than it should
> -----------------------------------------------
>
> Key: FALCON-321
> URL: https://issues.apache.org/jira/browse/FALCON-321
> Project: Falcon
> Issue Type: Bug
> Reporter: Raghav Kumar Gautam
> Priority: Blocker
> Labels: system-tests
>
> In FeedEvictor.java we have:
> {code:java}
> private void deleteParentIfEmpty(FileSystem fs, Path parent, Path
> feedBasePath) throws IOException {
> if (feedBasePath.equals(parent)) {
> LOG.info("Not deleting feed base path:" + parent);
> } else {
> if (fs.getContentSummary(parent).getFileCount() == 0) {
> LOG.info("Parent path: " + parent + " is empty, deleting
> path");
> if (fs.delete(parent, true)) {
> LOG.info("Deleted empty dir: " + parent);
> } else {
> throw new IOException("Unable to delete parent path:" +
> parent);
> }
> deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);
> }
> }
> }
> {code}
> In the fs.getContentSummary(parent).getFileCount() call if the parent has no
> files but has directories then we delete the parent directory. Which is
> incorrect.
> Here is log from falcon-regression's RetentionTest.testRetention(parameters:
> hours, 24, true, daily) :
> {noformat}
> 2014-02-24 15:09:45,034 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Applying retention on
> DATA=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}#META=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksMetaData#STATS=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksStats#TMP=/tmp
> type: instance, Limit: hours(24), timezone: UTC, frequency: hours,
> storageFILESYSTEM
> 2014-02-24 15:09:45,051 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Normalized path : /retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}
> 2014-02-24 15:09:45,123 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Searching for /retention/testFolders/*/*/*/*
> 2014-02-24 15:09:45,486 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted instance :/retention/testFolders/2014/01/21/00
> 2014-02-24 15:09:45,500 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Parent path: /retention/testFolders/2014/01/21 is empty, deleting path
> 2014-02-24 15:09:45,509 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted empty dir: /retention/testFolders/2014/01/21
> 2014-02-24 15:09:45,511 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Parent path: /retention/testFolders/2014/01 is empty, deleting path
> 2014-02-24 15:09:45,517 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted empty dir: /retention/testFolders/2014/01
> 2014-02-24 15:09:45,518 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Parent path: /retention/testFolders/2014 is empty, deleting path
> 2014-02-24 15:09:45,525 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Deleted empty dir: /retention/testFolders/2014
> 2014-02-24 15:09:45,526 INFO [main] org.apache.falcon.retention.FeedEvictor:
> Not deleting feed base path:/retention/testFolders
> {noformat}
> Stacktrace:
> {noformat}
> Failing Oozie Launcher, Main class [org.apache.falcon.retention.FeedEvictor],
> main() threw exception, Unable to delete instance:
> /retention/testFolders/2014/01/21/03
> java.io.IOException: Unable to delete instance:
> /retention/testFolders/2014/01/21/03
> at
> org.apache.falcon.retention.FeedEvictor.deleteInstance(FeedEvictor.java:321)
> at
> org.apache.falcon.retention.FeedEvictor.fileSystemEvictor(FeedEvictor.java:174)
> at org.apache.falcon.retention.FeedEvictor.evictFS(FeedEvictor.java:149)
> at org.apache.falcon.retention.FeedEvictor.evict(FeedEvictor.java:139)
> at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
