[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-04 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346652#comment-14346652
 ] 

Hadoop QA commented on YARN-3264:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12702408/YARN-3264.003.patch
  against trunk revision 3560180.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6835//console

This message is automatically generated.

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch, YARN-3264.002.patch, 
 YARN-3264.003.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-03 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346477#comment-14346477
 ] 

Robert Kanter commented on YARN-3264:
-

In addition to the things Zhijie said,

11. It would be good to refactor this to have the {{close}} in a finally block 
in case an Exception occurs we can clean up.
{code}
112   PrintWriter out = new PrintWriter(new BufferedWriter(new 
FileWriter(
113   fileName, true)));
114   out.println(TimelineUtils.dumpTimelineRecordtoJSON(entity));
115   out.write(\n);
116   out.close();
117 } catch (IOException ioe) {
{code}

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch, YARN-3264.002.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-03 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346212#comment-14346212
 ] 

Hadoop QA commented on YARN-3264:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12702338/YARN-3264.001.patch
  against trunk revision 5af693f.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6832//console

This message is automatically generated.

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-03 Thread Vrushali C (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346336#comment-14346336
 ] 

Vrushali C commented on YARN-3264:
--

[~zjshen]
Sure, we can use any name (or any more levels of nesting). I actually wanted to 
create a dir named by the name that gets passed in to the constructor in 
TimlineAggregator, but unable to figure out how to get the ReflectionUtils to 
create a new instance where something is passed in to the constructor. 

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-03 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346361#comment-14346361
 ] 

Hadoop QA commented on YARN-3264:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12702362/YARN-3264.002.patch
  against trunk revision f2d7a67.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6834//console

This message is automatically generated.

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch, YARN-3264.002.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-03 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346295#comment-14346295
 ] 

Zhijie Shen commented on YARN-3264:
---

Thanks for the patch, Vrushali! I'll review it. The storage scheme sounds good 
to me, but one quick comment: can we use some dir name such as 
timeline_service_data instead of the writer class name in the path?

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-03-03 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346355#comment-14346355
 ] 

Zhijie Shen commented on YARN-3264:
---

Overall, I think the patch should work. Some more detailed comments about the 
patch. Please take a look.

1. I think NO_START_TIME may no longer make sense in ATS v2.
{code}
98  /**
99   * Error code returned when no start time can be found when putting 
an
100  * entity. This occurs when the entity does not already exist in 
the store
101  * and it is put with no start time or events specified.
102  */
103 public static final int NO_START_TIME = 1;
{code}

2. For the POC implementation config, IMOH, it's better to be put in the writer 
impl. YarnConfiguration is the API realm. 
{code}
1253  /** Default PoC value for timeline service storage tmp root for FILE 
YARN-3264 */
1254  public static final String TIMELINE_SERVICE_STORAGE_DIR_ROOT = 
/tmp/;
{code}

3. Move the cleanup code into the finally block? Hence even the assertion 
failure will not leave uncleaned files.
{code}
359 // remove the files and dirs if empty
{code}

4. Make it private?
{code}
TimelineServiceWriter writer ;
{code}

5. Make it static? I suggest to have TIMELINE_SERVICE_STORAGE_ROOT_DIR as the 
config name = yarn.timeline-service.fs-writer.root-dir and have 
DEFAULT_TIMELINE_SERVICE_STORAGE_ROOT_DIR = /tmp/timeline_service_data, or 
something similar
{code}
private final String TMP_ROOT = /tmp/;
{code}

6. Some minor issue about class name and variable name
{code}
52 * are to be done are given by {@link AggregateUpTo}
{code}
{code}
65TimelineAggregationGranularity aggregateUpTo) throws IOException;
{code}

7. No need to declare JsonGenerationException and JsonMappingException.
{code}
101   TimelineServiceWriteResponse response) throws 
JsonGenerationException,
102   JsonMappingException, IOException {
{code}

8. Can we change it to debug level and put in a {{LOG.isDebugEnabled()}} block?
{code}
LOG.trace(SUCCESS - TIMELINE V2 PROTOTYPE);
90  LOG.trace(postEntities(entities= + entities + , callerUgi= +
79  callerUgi + ));   91  callerUgi + ));
{code}

9. Shall we also declare .thist as a private static final String?

10. You can use ConverterUtils to convert string to app/attempt/container id 
object, and you can construct or get app/attempt/container id object from each 
other. It may help you here to cleanup the code.
{code}
String appTimestampFileName = appId.replace(application_, appAttempt_);
{code}

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C
 Attachments: YARN-3264.001.patch, YARN-3264.002.patch


 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3264) [Storage implementation] Create a POC only file based storage implementation for ATS writes

2015-02-26 Thread Junping Du (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338584#comment-14338584
 ] 

Junping Du commented on YARN-3264:
--

+1. This is also useful for test.

 [Storage implementation] Create a POC only file based storage implementation 
 for ATS writes
 ---

 Key: YARN-3264
 URL: https://issues.apache.org/jira/browse/YARN-3264
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Vrushali C
Assignee: Vrushali C

 For the PoC, need to create a backend impl for file based storage of entities 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)