[
https://issues.apache.org/jira/browse/FALCON-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658741#comment-14658741
]
sandeep samudrala edited comment on FALCON-1368 at 8/5/15 7:41 PM:
-------------------------------------------------------------------
I have made changes to make configuration store loading of entities parallel
for each entity type in the respective order ( CLUSTER, FEED, PROCESS)
This is a sample of how it looks and there is a good dip in the restart time of
falcon server.
{noformat}
2015-08-06 00:59:04,687 INFO - [pool-5-thread-95:] ~ Restored configuration
FEED/ipfeed34 (ConfigurationStore:420)
2015-08-06 00:59:04,690 INFO - [pool-5-thread-94:] ~ Restored configuration
FEED/ipfeed18 (ConfigurationStore:420)
2015-08-06 00:59:04,694 INFO - [pool-5-thread-93:] ~ Restored configuration
FEED/ipfeed10 (ConfigurationStore:420)
{noformat}
was (Author: sandeep.samudrala):
I have made changes to make configuration store loading of entities parallel
for each entity type in the respective order ( CLUSTER, FEED, PROCESS)
This is a sample of how it looks and there is a huge dip in the restart time of
falcon server.
{noformat}
2015-08-06 00:59:04,687 INFO - [pool-5-thread-95:] ~ Restored configuration
FEED/ipfeed34 (ConfigurationStore:420)
2015-08-06 00:59:04,690 INFO - [pool-5-thread-94:] ~ Restored configuration
FEED/ipfeed18 (ConfigurationStore:420)
2015-08-06 00:59:04,694 INFO - [pool-5-thread-93:] ~ Restored configuration
FEED/ipfeed10 (ConfigurationStore:420)
{noformat}
> Improve Falcon server restart time
> -----------------------------------
>
> Key: FALCON-1368
> URL: https://issues.apache.org/jira/browse/FALCON-1368
> Project: Falcon
> Issue Type: Improvement
> Reporter: Ajay Yadava
> Assignee: sandeep samudrala
> Attachments: FALCON-1368.patch
>
>
> Currently on restart, Falcon server loads all the entities from HDFS one by
> one. In a large set up like the one at Inmobi, where we have several
> thousands of feeds and processes, this takes several minutes to increase the
> start up time.
> Since this is an IO intensive task(reading a file from HDFS into memory),
> having multiple threads to load entities in parallel will improve the start
> up time of the server.
> Two points need to be taken care of:
> 1. Only a single type of entities should be load in parallel to preserve the
> order of load of entities.
> 2. Currently Falcon server fails to start if there is an error in loading any
> entity, it will be slightly tricky but we should preserve the same behaviour
> in case of parallel uploads via threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
