GitHub user chtyim opened a pull request:
https://github.com/apache/twill/pull/21
(TWILL-63) Speed up application launch time
The general approach is better jar files management and to cache and reuse
jar files created through
class dependency tracing. The changes are further broken down as follows:
1. Refactor jars generation
- One jar containing the TwillLauncher (launcher.jar), created through
dependency tracing.
- This jar is the same for all applications.
- One jar containing all twill classes (twill.jar), created through
dependency tracing.
- This jar is the same for all applications.
- One jar containing the application class, created through dependency
tracing.
- This jar is generated based on the application being launched. It is
reusable when launching the same app multiple times.
- One jar containing user resources setup through TwillPreparer.
- This jar is not reused between apps.
- One jar containing runtime config needed by Twill
- logback.xml, jvm opts, environment, classpaths, ... etc
2. Let YARN to expand jars when localizing to containers instead of
expanding it programatically
- This save time in jar expansion when multiple containers are running on
the same host
3. Introduce a new configuration "twill.location.cache.dir" to enable jar
caching and reuse
- Currently only the launcher.jar, twill.jar and application jar will be
cached and reuse when possible
- Cache cleanup logic is also in place to remove files in cache directory
that is no longer used by application
4. The ApplicationBundler is improved to allow more flexible usage
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chtyim/twill feature/twill-63
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/twill/pull/21.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21
----
commit 786f3c6e075c15d929b43e322f8a869963a95b81
Author: Terence Yim <cht...@apache.org>
Date: 2016-12-07T01:05:11Z
(TWILL-63) Speed up application launch time
The general approach is better jar files management and to cache and reuse
jar files created through
class dependency tracing. The changes are further broken down as follows:
1. Refactor jars generation
- One jar containing the TwillLauncher (launcher.jar), created through
dependency tracing.
- This jar is the same for all applications.
- One jar containing all twill classes (twill.jar), created through
dependency tracing.
- This jar is the same for all applications.
- One jar containing the application class, created through dependency
tracing.
- This jar is generated based on the application being launched. It is
reusable when launching the same app multiple times.
- One jar containing user resources setup through TwillPreparer.
- This jar is not reused between apps.
- One jar containing runtime config needed by Twill
- logback.xml, jvm opts, environment, classpaths, ... etc
2. Let YARN to expand jars when localizing to containers instead of
expanding it programatically
- This save time in jar expansion when multiple containers are running on
the same host
3. Introduce a new configuration "twill.location.cache.dir" to enable jar
caching and reuse
- Currently only the launcher.jar, twill.jar and application jar will be
cached and reuse when possible
- Cache cleanup logic is also in place to remove files in cache directory
that is no longer used by application
4. The ApplicationBundler is improved to allow more flexible usage
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
