[GitHub] twill issue #21: (TWILL-63) Speed up application launch time

Github user chtyim commented on the issue: https://github.com/apache/twill/pull/21 Also rebased on `master` branch to resolve conflicts in `YarnTwillController`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] twill pull request #21: (TWILL-63) Speed up application launch time

Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/21#discussion_r95052538 --- Diff: twill-core/src/main/java/org/apache/twill/internal/ApplicationBundler.java --- @@ -97,10 +76,10 @@ public ApplicationBundler(Iterable

[jira] [Resolved] (TWILL-187) Max app start time should be configurable.

[ https://issues.apache.org/jira/browse/TWILL-187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terence Yim resolved TWILL-187. --- Resolution: Fixed > Max app start time should be configurable. >

[jira] [Commented] (TWILL-187) Max app start time should be configurable.

[ https://issues.apache.org/jira/browse/TWILL-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806799#comment-15806799 ] ASF GitHub Bot commented on TWILL-187: -- Github user asfgit closed the pull request at:

[GitHub] twill pull request #22: (TWILL-187) Added max start time

Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/22 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (TWILL-63) Speed up application launch time

[ https://issues.apache.org/jira/browse/TWILL-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806796#comment-15806796 ] ASF GitHub Bot commented on TWILL-63: - Github user chtyim commented on a diff in the pull request:

[GitHub] twill pull request #21: (TWILL-63) Speed up application launch time

Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/21#discussion_r95052416 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/LocationCacheCleaner.java --- @@ -0,0 +1,206 @@ +/* + * Licensed to the Apache Software

[jira] [Commented] (TWILL-63) Speed up application launch time

[ https://issues.apache.org/jira/browse/TWILL-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806593#comment-15806593 ] ASF GitHub Bot commented on TWILL-63: - Github user albertshau commented on a diff in the pull request:

[jira] [Commented] (TWILL-63) Speed up application launch time

[ https://issues.apache.org/jira/browse/TWILL-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806591#comment-15806591 ] ASF GitHub Bot commented on TWILL-63: - Github user albertshau commented on a diff in the pull request:

[GitHub] twill pull request #21: (TWILL-63) Speed up application launch time

Github user albertshau commented on a diff in the pull request: https://github.com/apache/twill/pull/21#discussion_r95048004 --- Diff: twill-core/src/main/java/org/apache/twill/internal/ApplicationBundler.java --- @@ -97,10 +76,10 @@ public ApplicationBundler(Iterable

[GitHub] twill pull request #21: (TWILL-63) Speed up application launch time

Github user albertshau commented on a diff in the pull request: https://github.com/apache/twill/pull/21#discussion_r95050388 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/LocationCacheCleaner.java --- @@ -0,0 +1,206 @@ +/* + * Licensed to the Apache Software

[GitHub] twill pull request #21: (TWILL-63) Speed up application launch time

Github user albertshau commented on a diff in the pull request: https://github.com/apache/twill/pull/21#discussion_r95050605 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillRunnerService.java --- @@ -277,9 +281,16 @@ public TwillPreparer prepare(TwillApplication

[jira] [Commented] (TWILL-187) Max app start time should be configurable.

[ https://issues.apache.org/jira/browse/TWILL-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806570#comment-15806570 ] ASF GitHub Bot commented on TWILL-187: -- Github user chtyim commented on a diff in the pull request:

[GitHub] twill pull request #22: (TWILL-187) Added max start time

Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/22#discussion_r95050430 --- Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java --- @@ -257,8 +258,19 @@ TwillPreparer setLogLevels(String runnableName,

[GitHub] twill issue #22: (TWILL-187) Added max start time

Github user chtyim commented on the issue: https://github.com/apache/twill/pull/22 For the stop timeout, it is cover by TWILL-192, as the changes are more complicate in the stop case --- If your project is set up for it, you can reply to this email and have your reply appear on

[jira] [Commented] (TWILL-187) Max app start time should be configurable.

[ https://issues.apache.org/jira/browse/TWILL-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806512#comment-15806512 ] ASF GitHub Bot commented on TWILL-187: -- Github user gokulavasan commented on a diff in the pull

[GitHub] twill pull request #22: (TWILL-187) Added max start time

Github user gokulavasan commented on a diff in the pull request: https://github.com/apache/twill/pull/22#discussion_r95049555 --- Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java --- @@ -257,8 +258,19 @@ TwillPreparer setLogLevels(String runnableName,

[jira] [Commented] (TWILL-187) Max app start time should be configurable.

[ https://issues.apache.org/jira/browse/TWILL-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806504#comment-15806504 ] ASF GitHub Bot commented on TWILL-187: -- GitHub user chtyim opened a pull request:

[GitHub] twill pull request #22: (TWILL-187) Added max start time

GitHub user chtyim opened a pull request: https://github.com/apache/twill/pull/22 (TWILL-187) Added max start time You can merge this pull request into a Git repository by running: $ git pull https://github.com/chtyim/twill feature/twill-187 Alternatively you can review and

Re: Desired behavior at shutdown

Terence, I see your point. I've thought a bit about this and it seems the only solution would be to coordinate via ZK between the controllers and the AM. The solution would be something like this: * Controller consumers register an ephemeral znode within under kafka znode to indicate they

[jira] [Updated] (TWILL-187) Max app start time should be configurable.

[ https://issues.apache.org/jira/browse/TWILL-187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terence Yim updated TWILL-187: -- Fix Version/s: 0.10.0 > Max app start time should be configurable. >

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805309#comment-15805309 ] ASF GitHub Bot commented on TWILL-199: -- Github user maochf commented on a diff in the pull request:

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user maochf commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r9505 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

Re: CPU count in Resource Spec

Yes, I did See attached On Fri, Jan 6, 2017 at 10:15 AM, Terence Yim wrote: > Hi, > > We've never observe this before. The AM container itself is always running > with 1 vcore. Have you look at the YARN container info in the RM UI that is > having that particular

Re: CPU count in Resource Spec

Hi, We've never observe this before. The AM container itself is always running with 1 vcore. Have you look at the YARN container info in the RM UI that is having that particular TwillRunnable running inside? Terence On Fri, Jan 6, 2017 at 9:54 AM, Yuliya Feldman wrote: >

Re: CPU count in Resource Spec

yes On Fri, Jan 6, 2017 at 9:40 AM, Terence Yim wrote: > Where do you verify the CPU cores? Was it from the YARN resource manager > UI? > > Terence > > Sent from my iPhone > > > On Jan 6, 2017, at 9:01 AM, Yuliya Feldman wrote: > > > > Sorry for so many

Re: CPU count in Resource Spec

Where do you verify the CPU cores? Was it from the YARN resource manager UI? Terence Sent from my iPhone > On Jan 6, 2017, at 9:01 AM, Yuliya Feldman wrote: > > Sorry for so many questions - unfortunately not much information I can fish > in the wild :). > > I noticed that

Re: logging within an application

The logging is controlled by logback xml configuration. Can you check if you have logback.xml or logback-test.xml in your classpath that might altered the behavior? Terence Sent from my iPhone > On Jan 6, 2017, at 8:54 AM, Yuliya Feldman wrote: > > Hello there, > > I

CPU count in Resource Spec

Sorry for so many questions - unfortunately not much information I can fish in the wild :). I noticed that CPU count set on Application Specification is not getting reflected while starting container - it is always 1 CPU. Did anybody experience this? It really feels like it is lost somewhere in

logging within an application

Hello there, I have an issue with logging from my App that runs within container - no logging comes out except one that goes to System.out Here is what I have in stderr of the container (not sure if StaticBinder duplicate is an issue, but I will certainly try to fix it): OpenJDK 64-Bit Server

Re: Retrieving failure when app terminates abnormally

Thank you Terrence On Fri, Jan 6, 2017 at 12:37 AM, Terence Yim wrote: > Hi, > > It depends on when the failure happen. If the app is already submitted to > YARN successfully and failed afterwards, it should still get reflected > through the Future returned by the teminate()

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804088#comment-15804088 ] ASF GitHub Bot commented on TWILL-199: -- Github user chtyim commented on a diff in the pull request:

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r94918807 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804030#comment-15804030 ] ASF GitHub Bot commented on TWILL-199: -- Github user maochf commented on a diff in the pull request:

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user maochf commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r94915705 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r94914909 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804014#comment-15804014 ] ASF GitHub Bot commented on TWILL-199: -- Github user chtyim commented on a diff in the pull request:

Re: Retrieving failure when app terminates abnormally

Hi, It depends on when the failure happen. If the app is already submitted to YARN successfully and failed afterwards, it should still get reflected through the Future returned by the teminate() call. However, due to https://issues.apache.org/jira/browse/TWILL-180

[jira] [Commented] (TWILL-180) TwillController.onTerminated() should pass the exit status of the app to the callback

[ https://issues.apache.org/jira/browse/TWILL-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804004#comment-15804004 ] Terence Yim commented on TWILL-180: --- Supposedly the failure can be get via the {{Future}} returned by the

[jira] [Updated] (TWILL-180) TwillController.onTerminated() should pass the exit status of the app to the callback

[ https://issues.apache.org/jira/browse/TWILL-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terence Yim updated TWILL-180: -- Fix Version/s: 0.10.0 > TwillController.onTerminated() should pass the exit status of the app to the >

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804001#comment-15804001 ] ASF GitHub Bot commented on TWILL-199: -- Github user maochf commented on a diff in the pull request:

[jira] [Assigned] (TWILL-180) TwillController.onTerminated() should pass the exit status of the app to the callback

[ https://issues.apache.org/jira/browse/TWILL-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terence Yim reassigned TWILL-180: - Assignee: Terence Yim > TwillController.onTerminated() should pass the exit status of the app to

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user maochf commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r94914331 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803993#comment-15803993 ] ASF GitHub Bot commented on TWILL-199: -- Github user chtyim commented on a diff in the pull request:

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803986#comment-15803986 ] ASF GitHub Bot commented on TWILL-199: -- Github user maochf commented on a diff in the pull request:

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user maochf commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r94913855 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803974#comment-15803974 ] ASF GitHub Bot commented on TWILL-199: -- Github user chtyim commented on a diff in the pull request:

[GitHub] twill pull request #16: [TWILL-199] Return the offset to read next message i...

Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/16#discussion_r94913560 --- Diff: twill-core/src/main/java/org/apache/twill/kafka/client/KafkaConsumer.java --- @@ -33,14 +33,16 @@ /** * Invoked when new

[jira] [Commented] (TWILL-199) Get next offset and handle offset error in KafkaConsumer.MessageCallback

[ https://issues.apache.org/jira/browse/TWILL-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803951#comment-15803951 ] ASF GitHub Bot commented on TWILL-199: -- Github user maochf commented on a diff in the pull request: