[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532991#comment-14532991 ] Xuan Gong commented on YARN-2137: - close this as duplicate. Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Assignee: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527522#comment-14527522 ] Xuan Gong commented on YARN-2137: - [~ksumit] I think that https://issues.apache.org/jira/browse/YARN-3269 might be duplicate for this issue. Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Assignee: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524303#comment-14524303 ] Xuan Gong commented on YARN-2137: - [~ksumit] Any update for this one ? Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Assignee: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524511#comment-14524511 ] Sumit Kumar commented on YARN-2137: --- Apologies for the delay, i will rebase this patch and look into the required testing that [~vinodkv] recommended. Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Assignee: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14029436#comment-14029436 ] Sumit Kumar commented on YARN-2137: --- Thanks for pointing these out [~vinodkv]. These make sense :-) working on these right now. Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Assignee: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[
https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025689#comment-14025689
]
Hadoop QA commented on YARN-2137:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12649443/YARN-2137.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3946//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3946//console
This message is automatically generated.
Add support for logaggregation to a path on non-default filecontext
---
Key: YARN-2137
URL: https://issues.apache.org/jira/browse/YARN-2137
Project: Hadoop YARN
Issue Type: Improvement
Components: log-aggregation
Affects Versions: 2.4.0
Reporter: Sumit Kumar
Attachments: YARN-2137.patch
Current log-aggregation implementation supports logaggregation to default
filecontext only. This patch is to support logaggregation to any of the
supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So
for example a customer could use hdfs as default filesystem but use s3 or
swiftfs for logaggregation. Current implementation makes mixed usages of
FileContext+AbstractFileSystem apis as well as FileSystem apis which is
confusing.
This patch does two things:
# moves logaggregation implementation to use only FileContext apis
# adds support for doing log aggregation on non-default filesystem as well.
# changes TestLogAggregationService to use local filesystem itself instead of
mocking the behavior
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026055#comment-14026055 ] Varun Vasudev commented on YARN-2137: - Sumit, have you tested this with the client? There is a yarn logs command that dumps the logs give an application id. Does the client automatically pick up the right FileContext? Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026113#comment-14026113 ] Sumit Kumar commented on YARN-2137: --- [~vvasudev] i tested with the unit test only. if you think testing with yarn logs command adds more robustness, will do so as well. Please let me know. Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2137) Add support for logaggregation to a path on non-default filecontext
[ https://issues.apache.org/jira/browse/YARN-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026116#comment-14026116 ] Varun Vasudev commented on YARN-2137: - No sorry. I should have been more clear. http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#logs - the yarn logs command is used to dump the logs that you aggregate on the remote log dir. Have you confirmed that it uses the right FileContext? Add support for logaggregation to a path on non-default filecontext --- Key: YARN-2137 URL: https://issues.apache.org/jira/browse/YARN-2137 Project: Hadoop YARN Issue Type: Improvement Components: log-aggregation Affects Versions: 2.4.0 Reporter: Sumit Kumar Attachments: YARN-2137.patch Current log-aggregation implementation supports logaggregation to default filecontext only. This patch is to support logaggregation to any of the supported filesystems within hadoop eco-system (hdfs, s3, swiftfs etc). So for example a customer could use hdfs as default filesystem but use s3 or swiftfs for logaggregation. Current implementation makes mixed usages of FileContext+AbstractFileSystem apis as well as FileSystem apis which is confusing. This patch does two things: # moves logaggregation implementation to use only FileContext apis # adds support for doing log aggregation on non-default filesystem as well. # changes TestLogAggregationService to use local filesystem itself instead of mocking the behavior -- This message was sent by Atlassian JIRA (v6.2#6252)
