[jira] [Commented] (TWILL-122) Allow disabling the log transport
[ https://issues.apache.org/jira/browse/TWILL-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935331#comment-15935331 ] ASF GitHub Bot commented on TWILL-122: -- GitHub user chtyim opened a pull request: https://github.com/apache/twill/pull/40 (TWILL-122) Allow disabling log collection - Introduced a new configuration twill.log.collection.enabled for turning off log collection - Refactor YarnTwillController and related class hierarchy to not starting Kafka client when log collection is disabled - Added Kafka zk connection string information in AM live node data - Refactor KafkaAppender and ServiceMain configureLogger - Log to StatusManager instead of Logger to avoid recursive logging - Instead of resetting logback configuration, directly instantiate and add the Kafka log appender to the logging context. - Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to simplify ZK Connection string construction You can merge this pull request into a Git repository by running: $ git pull https://github.com/chtyim/twill feature/TWILL-122 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/40.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #40 commit 617280ee72b2e7fc7a3cc473d6dfe35576cf5e57 Author: Terence Yim Date: 2017-03-20T21:42:46Z (TWILL-122) Allow disabling log collection - Introduced a new configuration twill.log.collection.enabled for turning off log collection - Refactor YarnTwillController and related class hierarchy to not starting Kafka client when log collection is disabled - Added Kafka zk connection string information in AM live node data - Refactor KafkaAppender and ServiceMain configureLogger - Log to StatusManager instead of Logger to avoid recursive logging - Instead of resetting logback configuration, directly instantiate and add the Kafka log appender to the logging context. - Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to simplify ZK Connection string construction > Allow disabling the log transport > - > > Key: TWILL-122 > URL: https://issues.apache.org/jira/browse/TWILL-122 > Project: Apache Twill > Issue Type: Improvement > Components: core >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently transporting logs to Kafka is mandatory. It should be optionally so > that application can have its own way of log collection. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #40: (TWILL-122) Allow disabling log collection
GitHub user chtyim opened a pull request: https://github.com/apache/twill/pull/40 (TWILL-122) Allow disabling log collection - Introduced a new configuration twill.log.collection.enabled for turning off log collection - Refactor YarnTwillController and related class hierarchy to not starting Kafka client when log collection is disabled - Added Kafka zk connection string information in AM live node data - Refactor KafkaAppender and ServiceMain configureLogger - Log to StatusManager instead of Logger to avoid recursive logging - Instead of resetting logback configuration, directly instantiate and add the Kafka log appender to the logging context. - Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to simplify ZK Connection string construction You can merge this pull request into a Git repository by running: $ git pull https://github.com/chtyim/twill feature/TWILL-122 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/40.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #40 commit 617280ee72b2e7fc7a3cc473d6dfe35576cf5e57 Author: Terence Yim Date: 2017-03-20T21:42:46Z (TWILL-122) Allow disabling log collection - Introduced a new configuration twill.log.collection.enabled for turning off log collection - Refactor YarnTwillController and related class hierarchy to not starting Kafka client when log collection is disabled - Added Kafka zk connection string information in AM live node data - Refactor KafkaAppender and ServiceMain configureLogger - Log to StatusManager instead of Logger to avoid recursive logging - Instead of resetting logback configuration, directly instantiate and add the Kafka log appender to the logging context. - Refactor ServiceMain, ApplicationMasterMain and TwillContainerMain to simplify ZK Connection string construction --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (TWILL-225) Allow using different configurations per application submission
[ https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terence Yim resolved TWILL-225. --- Resolution: Fixed > Allow using different configurations per application submission > --- > > Key: TWILL-225 > URL: https://issues.apache.org/jira/browse/TWILL-225 > Project: Apache Twill > Issue Type: Improvement >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently there are couple configurations that can be provided via the hadoop > {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those > configurations are global (same for all app launched through the same > {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes > method to alter the configuration for a given app submission. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TWILL-225) Allow using different configurations per application submission
[ https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935282#comment-15935282 ] ASF GitHub Bot commented on TWILL-225: -- Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/39 > Allow using different configurations per application submission > --- > > Key: TWILL-225 > URL: https://issues.apache.org/jira/browse/TWILL-225 > Project: Apache Twill > Issue Type: Improvement >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently there are couple configurations that can be provided via the hadoop > {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those > configurations are global (same for all app launched through the same > {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes > method to alter the configuration for a given app submission. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...
Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/39 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-225) Allow using different configurations per application submission
[ https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935255#comment-15935255 ] ASF GitHub Bot commented on TWILL-225: -- Github user anwar6953 commented on a diff in the pull request: https://github.com/apache/twill/pull/39#discussion_r107265236 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java --- @@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, Map logL this.logLevels.put(runnableName, newLevels); } + /** + * Creates an {@link Credentials} by copying the {@link Credentials} of the current user. + */ private Credentials createCredentials() { Credentials credentials = new Credentials(); try { credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials()); +} catch (IOException e) { --- End diff -- I guess app submission would fail if caller doesn't acquire required credentials in its own way anyways. > Allow using different configurations per application submission > --- > > Key: TWILL-225 > URL: https://issues.apache.org/jira/browse/TWILL-225 > Project: Apache Twill > Issue Type: Improvement >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently there are couple configurations that can be provided via the hadoop > {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those > configurations are global (same for all app launched through the same > {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes > method to alter the configuration for a given app submission. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TWILL-225) Allow using different configurations per application submission
[ https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935253#comment-15935253 ] ASF GitHub Bot commented on TWILL-225: -- Github user anwar6953 commented on the issue: https://github.com/apache/twill/pull/39 LGTM > Allow using different configurations per application submission > --- > > Key: TWILL-225 > URL: https://issues.apache.org/jira/browse/TWILL-225 > Project: Apache Twill > Issue Type: Improvement >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently there are couple configurations that can be provided via the hadoop > {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those > configurations are global (same for all app launched through the same > {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes > method to alter the configuration for a given app submission. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...
Github user anwar6953 commented on a diff in the pull request: https://github.com/apache/twill/pull/39#discussion_r107265236 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java --- @@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, Map logL this.logLevels.put(runnableName, newLevels); } + /** + * Creates an {@link Credentials} by copying the {@link Credentials} of the current user. + */ private Credentials createCredentials() { Credentials credentials = new Credentials(); try { credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials()); +} catch (IOException e) { --- End diff -- I guess app submission would fail if caller doesn't acquire required credentials in its own way anyways. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #39: (TWILL-225) Refactor TwillPreprer to allow configurations a...
Github user anwar6953 commented on the issue: https://github.com/apache/twill/pull/39 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-225) Allow using different configurations per application submission
[ https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935247#comment-15935247 ] ASF GitHub Bot commented on TWILL-225: -- Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/39#discussion_r107264780 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java --- @@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, Map logL this.logLevels.put(runnableName, newLevels); } + /** + * Creates an {@link Credentials} by copying the {@link Credentials} of the current user. + */ private Credentials createCredentials() { Credentials credentials = new Credentials(); try { credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials()); +} catch (IOException e) { --- End diff -- I don't think so. Even if this call fail, if the caller is able to acquire required credentials in his own way and add them via the `addSecureStore` method, then the app can still be submitted and executed correctly. > Allow using different configurations per application submission > --- > > Key: TWILL-225 > URL: https://issues.apache.org/jira/browse/TWILL-225 > Project: Apache Twill > Issue Type: Improvement >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently there are couple configurations that can be provided via the hadoop > {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those > configurations are global (same for all app launched through the same > {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes > method to alter the configuration for a given app submission. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...
Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/39#discussion_r107264780 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java --- @@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, Map logL this.logLevels.put(runnableName, newLevels); } + /** + * Creates an {@link Credentials} by copying the {@link Credentials} of the current user. + */ private Credentials createCredentials() { Credentials credentials = new Credentials(); try { credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials()); +} catch (IOException e) { --- End diff -- I don't think so. Even if this call fail, if the caller is able to acquire required credentials in his own way and add them via the `addSecureStore` method, then the app can still be submitted and executed correctly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-225) Allow using different configurations per application submission
[ https://issues.apache.org/jira/browse/TWILL-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935227#comment-15935227 ] ASF GitHub Bot commented on TWILL-225: -- Github user anwar6953 commented on a diff in the pull request: https://github.com/apache/twill/pull/39#discussion_r107262119 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java --- @@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, Map logL this.logLevels.put(runnableName, newLevels); } + /** + * Creates an {@link Credentials} by copying the {@link Credentials} of the current user. + */ private Credentials createCredentials() { Credentials credentials = new Credentials(); try { credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials()); +} catch (IOException e) { --- End diff -- shouldn't this be fatal? > Allow using different configurations per application submission > --- > > Key: TWILL-225 > URL: https://issues.apache.org/jira/browse/TWILL-225 > Project: Apache Twill > Issue Type: Improvement >Reporter: Terence Yim >Assignee: Terence Yim > Fix For: 0.11.0 > > > Currently there are couple configurations that can be provided via the hadoop > {{Configuration}} object to the {{YarnTwillRunnerService}}. However, those > configurations are global (same for all app launched through the same > {{TwillRunnerService}}). It would be better if the {{TwillPreparer}} exposes > method to alter the configuration for a given app submission. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #39: (TWILL-225) Refactor TwillPreprer to allow configura...
Github user anwar6953 commented on a diff in the pull request: https://github.com/apache/twill/pull/39#discussion_r107262119 --- Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java --- @@ -455,21 +475,40 @@ private void saveLogLevels(String runnableName, Map logL this.logLevels.put(runnableName, newLevels); } + /** + * Creates an {@link Credentials} by copying the {@link Credentials} of the current user. + */ private Credentials createCredentials() { Credentials credentials = new Credentials(); try { credentials.addAll(UserGroupInformation.getCurrentUser().getCredentials()); +} catch (IOException e) { --- End diff -- shouldn't this be fatal? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---