[jira] [Commented] (YARN-2519) Credential Provider related unit tests failed on Windows
[
https://issues.apache.org/jira/browse/YARN-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124867#comment-14124867
]
Hudson commented on YARN-2519:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #673 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/673/])
YARN-2519. Credential Provider related unit tests failed on Windows.
Contributed by Xiaoyu Yao. (cnauroth: rev
cbea1b10efd871d04c648af18449dc724685db74)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/util/TestWebAppUtils.java
Credential Provider related unit tests failed on Windows
Key: YARN-2519
URL: https://issues.apache.org/jira/browse/YARN-2519
Project: Hadoop YARN
Issue Type: Test
Components: webapp
Affects Versions: 2.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Fix For: 2.6.0
Attachments: YARN-2519.0.patch
Reported by: Xiaomara and investigated by Chris Nauroth.
The credential provider related unit tests failed on Windows. The tests try
to set up a URI by taking the build test directory and concatenating it with
other strings containing the rest of the URI format, i.e.:
{code}
public void testFactory() throws Exception {
Configuration conf = new Configuration();
conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
UserProvider.SCHEME_NAME + :///, +
JavaKeyStoreProvider.SCHEME_NAME + ://file + tmpDir +
/test.jks);
{code}
This logic is incorrect on Windows, because the file path separator will be
'\', which violates URI syntax. Forward slash is not permitted.
The proper fix is to always do path/URI construction through the
org.apache.hadoop.fs.Path class, specifically using the constructors that
take explicit parent and child arguments.
The affected unit tests are:
{code}
* TestWebAppUtils
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2519) Credential Provider related unit tests failed on Windows
[
https://issues.apache.org/jira/browse/YARN-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124899#comment-14124899
]
Hudson commented on YARN-2519:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1864 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1864/])
YARN-2519. Credential Provider related unit tests failed on Windows.
Contributed by Xiaoyu Yao. (cnauroth: rev
cbea1b10efd871d04c648af18449dc724685db74)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/util/TestWebAppUtils.java
* hadoop-yarn-project/CHANGES.txt
Credential Provider related unit tests failed on Windows
Key: YARN-2519
URL: https://issues.apache.org/jira/browse/YARN-2519
Project: Hadoop YARN
Issue Type: Test
Components: webapp
Affects Versions: 2.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Fix For: 2.6.0
Attachments: YARN-2519.0.patch
Reported by: Xiaomara and investigated by Chris Nauroth.
The credential provider related unit tests failed on Windows. The tests try
to set up a URI by taking the build test directory and concatenating it with
other strings containing the rest of the URI format, i.e.:
{code}
public void testFactory() throws Exception {
Configuration conf = new Configuration();
conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
UserProvider.SCHEME_NAME + :///, +
JavaKeyStoreProvider.SCHEME_NAME + ://file + tmpDir +
/test.jks);
{code}
This logic is incorrect on Windows, because the file path separator will be
'\', which violates URI syntax. Forward slash is not permitted.
The proper fix is to always do path/URI construction through the
org.apache.hadoop.fs.Path class, specifically using the constructors that
take explicit parent and child arguments.
The affected unit tests are:
{code}
* TestWebAppUtils
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2519) Credential Provider related unit tests failed on Windows
[
https://issues.apache.org/jira/browse/YARN-2519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124907#comment-14124907
]
Hudson commented on YARN-2519:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1889 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1889/])
YARN-2519. Credential Provider related unit tests failed on Windows.
Contributed by Xiaoyu Yao. (cnauroth: rev
cbea1b10efd871d04c648af18449dc724685db74)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/util/TestWebAppUtils.java
Credential Provider related unit tests failed on Windows
Key: YARN-2519
URL: https://issues.apache.org/jira/browse/YARN-2519
Project: Hadoop YARN
Issue Type: Test
Components: webapp
Affects Versions: 2.4.1
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Fix For: 2.6.0
Attachments: YARN-2519.0.patch
Reported by: Xiaomara and investigated by Chris Nauroth.
The credential provider related unit tests failed on Windows. The tests try
to set up a URI by taking the build test directory and concatenating it with
other strings containing the rest of the URI format, i.e.:
{code}
public void testFactory() throws Exception {
Configuration conf = new Configuration();
conf.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH,
UserProvider.SCHEME_NAME + :///, +
JavaKeyStoreProvider.SCHEME_NAME + ://file + tmpDir +
/test.jks);
{code}
This logic is incorrect on Windows, because the file path separator will be
'\', which violates URI syntax. Forward slash is not permitted.
The proper fix is to always do path/URI construction through the
org.apache.hadoop.fs.Path class, specifically using the constructors that
take explicit parent and child arguments.
The affected unit tests are:
{code}
* TestWebAppUtils
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1621) Add CLI to list rows of task attempt ID, container ID, host of container, state of container
[
https://issues.apache.org/jira/browse/YARN-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125023#comment-14125023
]
Hadoop QA commented on YARN-1621:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667097/YARN-1621.1.patch
against trunk revision a23144f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4841//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4841//console
This message is automatically generated.
Add CLI to list rows of task attempt ID, container ID, host of container,
state of container
--
Key: YARN-1621
URL: https://issues.apache.org/jira/browse/YARN-1621
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.2.0
Reporter: Tassapol Athiapinya
Fix For: 2.6.0
Attachments: YARN-1621.1.patch
As more applications are moved to YARN, we need generic CLI to list rows of
task attempt ID, container ID, host of container, state of container. Today
if YARN application running in a container does hang, there is no way to find
out more info because a user does not know where each attempt is running in.
For each running application, it is useful to differentiate between
running/succeeded/failed/killed containers.
{code:title=proposed yarn cli}
$ yarn application -list-containers -applicationId appId [-containerState
state of container]
where containerState is optional filter to list container in given state only.
container state can be running/succeeded/killed/failed/all.
A user can specify more than one container state at once e.g. KILLED,FAILED.
task attempt ID container ID host of container state of container
{code}
CLI should work with running application/completed application. If a
container runs many task attempts, all attempts should be shown. That will
likely be the case of Tez container-reuse application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2410) Nodemanager ShuffleHandler can easily exhaust file descriptors
[ https://issues.apache.org/jira/browse/YARN-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125094#comment-14125094 ] Chen He commented on YARN-2410: --- I propose to use ChannelPipe to replace current ChannelFuture. Nodemanager ShuffleHandler can easily exhaust file descriptors -- Key: YARN-2410 URL: https://issues.apache.org/jira/browse/YARN-2410 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.5.0 Reporter: Nathan Roberts Assignee: Chen He Priority: Critical The async nature of the shufflehandler can cause it to open a huge number of file descriptors, when it runs out it crashes. Scenario: Job with 6K reduces, slow start set to 0.95, about 40 map outputs per node. Let's say all 6K reduces hit a node at about same time asking for their outputs. Each reducer will ask for all 40 map outputs over a single socket in a single request (not necessarily all 40 at once, but with coalescing it is likely to be a large number). sendMapOutput() will open the file for random reading and then perform an async transfer of the particular portion of this file(). This will theoretically happen 6000*40=24 times which will run the NM out of file descriptors and cause it to crash. The algorithm should be refactored a little to not open the fds until they're actually needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2512) Allow for origin pattern matching in cross origin filter
[ https://issues.apache.org/jira/browse/YARN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125113#comment-14125113 ] Zhijie Shen commented on YARN-2512: --- bq. I think there could be some confusion since URLs have a different RFC than hostnames and domains. My bad, I think you're right about the host name rule. bq. I'm slowly bringing hadoop up to feature parity with jetty so that users will have a full feature set and make the transition to jetty seamless when we move to jetty 7+. Sounds a good plan. +1, I'll commit the patch. Allow for origin pattern matching in cross origin filter Key: YARN-2512 URL: https://issues.apache.org/jira/browse/YARN-2512 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: YARN-2512-v1.patch Extending the feature set of allowed origins. Now a * in a pattern indicates this allowed origin is a pattern and will be matched including multiple sub-domains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125121#comment-14125121
]
Yuliya Feldman commented on YARN-2493:
--
to [~wangda]
I did not quite understand what is the purpose of
{code} ResourceRequest getAMContainerResourceRequest {code}
I can see in the above comment you are talking about the difference in ability
to specify number of containers in regular request unlike in AMRequest, so is
this new method to consolidate Priority, Resource into
AMContainerResourceRequest?
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125128#comment-14125128
]
Wangda Tan commented on YARN-2493:
--
Hi [~yufeldman],
Thanks for reviewing this, the purpose of add an AMContainerResourceRequest is
exactly as you said: Currently we have Priority, Resource, we're adding
labelExpression, in the future we will add resourceName and relaxLocality --
that is pretty much ResourceRequest, except numContainers, we don't need
duplicate them in ApplicationSubmissionContext.
Does this make sense to you?
Wangda
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125132#comment-14125132
]
Yuliya Feldman commented on YARN-2493:
--
[~wangda],
In this case why do you even bother adding labelExpression to
ApplicationSubmissionContext if you plan to have it as part of
AMResourceRequest, unless you want to keep it in both places for some reason?
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125133#comment-14125133
]
Wangda Tan commented on YARN-2493:
--
[~yufeldman],
The labelExpression added to ApplicationSubmissionContext is used to app-level
label expression, which will automatically set labelExpression for all
ResourceRequest (User can overwrite this by setting labelExpression in
individual ResourceRequest.
And the labelExpression of AMResourceRequest is only to set labelExpression on
AM container.
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125135#comment-14125135
]
Yuliya Feldman commented on YARN-2493:
--
[~leftnoteasy]]
OK, so what is the logic going to be:
1. If label is specified in ApplicationSubmisisonContext and AMResourceRequest
the one from former will be set for all the ResourceRequests except
AMResourceRequest (and can be overwritten later on), is it true?
2. If label is specified in ApplicationSubmisisonContext and not in
AMResourceRequest it will be set for all resource requests including
AMResourceRequest, is it true?
3. If label is not specified in ApplicationSubmisisonContext and specified in
AMResourceRequest - then what - same as #2?
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125139#comment-14125139
]
Wangda Tan commented on YARN-2493:
--
1/2 are true,
For 3, only AM container will be set the labelExpression.
Do you think is this easy/straightforward to use in your perspective? Do you
have any suggestions?
Thanks,
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125146#comment-14125146
]
Yuliya Feldman commented on YARN-2493:
--
[~leftnoteasy]
I suspect that #2 will be the most common, since either somebody will set label
for whole application, or will set label for non-AM container resource requests
programmatically (kind of #3). I think I saw in the comments: ability to set it
separately for mappers and reducers.
In my personal opinion it is better to at least restrict input parameters to
one (not two), otherwise we would need to take care of 4 combinations instead
of just 1.
In any case I am fine with the changes.
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2493) [YARN-796] API changes for users
[
https://issues.apache.org/jira/browse/YARN-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125163#comment-14125163
]
Wangda Tan commented on YARN-2493:
--
[~yufeldman],
Thanks for the suggestion,
I agree with you that adding am/app labelExpression separately is harder for
user to set (4 combinations instead of one). But the downside is we treat AM
container separately (we cannot only set label for AM container request).
Assume a user want to set label of AM = x, label of non-AM = , he will set
label expression of app=x, and label expression of all other
ResourceRequests=. It's not very convenient for user if he wants to set label
expression of AM container different from non-AM containers.
I'm fine with your suggestions if most use cases don't need set label
expression of AM container different from non-AM containers. Let's wait more
suggestions from community.
Thanks,
Wangda
[YARN-796] API changes for users
Key: YARN-2493
URL: https://issues.apache.org/jira/browse/YARN-2493
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-2493.patch, YARN-2493.patch
This JIRA includes API changes for users of YARN-796, like changes in
{{ResourceRequest}}, {{ApplicationSubmissionContext}}, etc. This is a common
part of YARN-796.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2284) Find missing config options in YarnConfiguration and yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125185#comment-14125185 ] Ray Chiang commented on YARN-2284: -- RE: Failing unit test. Works fine in my tree. Doesn't appear to be related to code changes. Find missing config options in YarnConfiguration and yarn-default.xml - Key: YARN-2284 URL: https://issues.apache.org/jira/browse/YARN-2284 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.4.1 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: YARN-2284-04.patch, YARN-2284-05.patch, YARN-2284-06.patch, YARN-2284-07.patch, YARN2284-01.patch, YARN2284-02.patch, YARN2284-03.patch YarnConfiguration has one set of properties. yarn-default.xml has another set of properties. Ideally, there should be an automatic way to find missing properties in either location. This is analogous to MAPREDUCE-5130, but for yarn-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1530) [Umbrella] Store, manage and serve per-framework application-timeline data
[ https://issues.apache.org/jira/browse/YARN-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125238#comment-14125238 ] Zhijie Shen commented on YARN-1530: --- [~rkanter], thanks for proposing the idea of pipeline writing, which sounds interesting. W.R.T the two proposed solutions, I have some general comments. 1. HDFS-based implementation: It sounds an interesting idea of using HDFS to persist unpublished timeline entities, but I’m not sure it’s going to solve the scalability problem. Though each application can write the timeline entities into HDFS in a distributed manner, there’s still a single timeline server that fetches the files of the timeline entities written by ALL applications. The bottleneck is still there. Essentially I don’t see any difference between publishing entities via HTTP REST interface and via HDFS in terms of scalability. And given the same timeline sever, I’m afraid that HTTP REST interface is very likely to be be more efficient: a) Less I/O against the secondary storage; b) Bult-in multithreading mechanism from the web server; c) No delay of waiting until next round of file fetching. The current writing channel allows the data to be available on the timeline server immediately, such that we can support the realtime or near realtime query about the application metrics. Because of c), I’m not sure this is still feasible with HDFS writing channel (too frequent fetching will have performance problem?). IMHO, the ESSENTIAL problem is that a single instance of the timeline server is not going to to intake a huge number of concurrent requests (no matter they’re from HTTP REST, RPC or HDFS). Let’s assume we’re going to have the HBase timeline store (YARN-2032), which is scalable and reliable. Thanks to the stateless nature, the proper way scale up the timeline service to start a number of federal timeline sever and connect them to the same HBase timeline store. This design will solve the high availability problem as well. Moreover, it’s not just scale the writing channel, but also the reading one. I’ll file a separate ticket about the scalable and highly available timeline server. 2. Direct-writing implementation: The biggest concern here is that the solution is cracking the current timeline server architecture: a) Accepting requests for users - b) Pre-prosessing the timeline entities and verifying users’ access - c) Transmitting the timeline entities into key/value pairs If you want to open the option for the client to write into the data store directly, all this logic has to be moved to the client. It’s a complex stack than some simple put calls to the data store. It’s not just increasing the more dependency for the timeline client, but changing it from a thin client to a fat one. This makes it more difficult to distribute the timeline client and upgrade it in the future, and makes the user heavily loaded. Importantly, I’m not sure we will be able to verify user's access at the client side, or HBase access control is enough for our specific timeline ACLs. Another concern is that the client is strongly coupled with a particular type of data store, such as HBase. If we choose to use Leveldb (or Rocksdb), the current client for HBase used at the application side is going to be broken. In other word, you have to change the client and the data store simultaneously. Misc. SLA of TimelineClient The proposal recalls me another interesting question: SLA of the timeline server and the client. The timeline server should be reliable: when an timeline entity is accepted at the timeline server, it should not be lost, which is ensured by reliable data store (e.g. HBase): YARN-2032. It is questionable whether we also want the timeline client to reliable: when an timeline entity is passed to the timeline client, it should not be lost before being accepted by the timeline server, which may be in the outage for a while. Hence, local cache in HDFS may be a good idea. Or we can use an even lightweight solution: Leveldb. I’ll file a ticket about it as well. [Umbrella] Store, manage and serve per-framework application-timeline data -- Key: YARN-1530 URL: https://issues.apache.org/jira/browse/YARN-1530 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Attachments: ATS-Write-Pipeline-Design-Proposal.pdf, ATS-meet-up-8-28-2014-notes.pdf, application timeline design-20140108.pdf, application timeline design-20140116.pdf, application timeline design-20140130.pdf, application timeline design-20140210.pdf This is a sibling JIRA for YARN-321. Today, each application/framework has to do store, and serve per-framework data all by itself as YARN doesn't have a common solution. This JIRA attempts to
