[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2016-07-10 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369733#comment-15369733
 ] 

Hudson commented on YARN-3391:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10074/])
YARN-3391. Clearly define flow ID/ flow run / flow version in API and (sjlee: 
rev 47f35a30bb4d99349593e9d6e1c9e76e71341c40)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestPerNodeTimelineCollectorsAuxService.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/GetTimelineCollectorContextResponsePBImpl.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/collectormanager/NMCollectorService.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollectorManager.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineWriterImpl.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestFileSystemTimelineWriterImpl.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/GetTimelineCollectorContextResponse.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/AppLevelTimelineCollector.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/Application.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockApp.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/TimelineWriter.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
* 

[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-14 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494915#comment-14494915
 ] 

Sangjin Lee commented on YARN-3391:
---

Thanks [~zjshen] for the patch!

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: YARN-2928

 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch, YARN-3391.5.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-09 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487314#comment-14487314
 ] 

Junping Du commented on YARN-3391:
--

Thanks [~vrushalic] for review, v5 patch LGTM too. [~vinodkv], any additional 
comments?

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch, YARN-3391.5.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-09 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487898#comment-14487898
 ] 

Junping Du commented on YARN-3391:
--

Sync offline with Vinod that he is fine with the latest patch. I will go ahead 
to commit it soon.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch, YARN-3391.5.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-08 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484815#comment-14484815
 ] 

Zhijie Shen commented on YARN-3391:
---

I created a new patch:

bq.  So in general, I think we should use as much javadoc comments instead of 
inline comments for public APIs.

Move the comments into TimelineUtils and make them javadoc.

bq. We should add more info to LOG.warn messages, at least to tell user flow 
run should be numeric.

Improve the warn message

bq. In addition, do we need to check negative value for flow run here?

According to Sangjin's given example, we usually want to identify a flow run by 
timestamp, which theoretically can be negative to represent sometime before 
1970.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-08 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485237#comment-14485237
 ] 

Junping Du commented on YARN-3391:
--

Thanks [~zjshen] for updating the patch!
bq. According to Sangjin's given example, we usually want to identify a flow 
run by timestamp, which theoretically can be negative to represent sometime 
before 1970.
Except time travel, I don't believe any flow run running on hadoop and new 
timeline service should happen before 1970. :) 
Anyway, we do have some practice to check timestamp  0 (like: 
MetricsRecordImpl), but more cases sounds like we didn't do this negative check 
for timestamp. Given this, I am fine with not checking here.

v4 patch looks good to me. [~sjlee0], [~vrushalic] and [~jrottinghuis], any 
additional comments for the patch?

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-08 Thread Vinod Kumar Vavilapalli (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485781#comment-14485781
 ] 

Vinod Kumar Vavilapalli commented on YARN-3391:
---

A cosmetic suggestion: flow_run - flow_run_name or flow_run_id ?

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-08 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485791#comment-14485791
 ] 

Vrushali C commented on YARN-3391:
--

[~vinodkv] ,
+1 for flow_run to be called as flow_run_id. 
It's a number (epoch timestamp). If we call it flow_run_name, that makes it 
sound like it's a string. 

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-08 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486068#comment-14486068
 ] 

Vrushali C commented on YARN-3391:
--

thanks [~zjshen], patch looks good. 

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch, 
 YARN-3391.4.patch, YARN-3391.5.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-07 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483498#comment-14483498
 ] 

Junping Du commented on YARN-3391:
--

Sorry for coming a little late. Thanks guys for good discussions here and 
[~zjshen] for updating the patch!
bq. I just wanted to add my 2 cents that this is something we already see and 
experience with hRaven so it's not theoretical.
+1, [~sjlee0]! I think that's very important feedback for improving user 
experience for new feature here. Let's try to get a good balance between 
addressing these solid scenarios as well as providing flexibility to possible 
new scenarios. e.g. we can provide different flow group policies that user can 
use to group application into flow by name or keeping them as isolated flow, 
etc. Anyway, as everyone's agreement so far, let's continue the discussion on a 
separated JIRA for figuring it out later. 

The patch looks good in overall. However, I still haven't seen we put 
definition of flow, flow run and flow version in any places of Javadoc. 
As I mentioned earlier, it should be useful for developers. The official Apache 
feature doc is more user oriented and we can address it later when feature get 
completed. 



 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-07 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483812#comment-14483812
 ] 

Zhijie Shen commented on YARN-3391:
---

bq. let's continue the discussion on a separated JIRA for figuring it out later.

Agree. Let's unblock this Jira which will unblock the writer implementation 
consequently. I filed YARN-3461 to continue the defaults discussion there. 

bq. I just wanted to add my 2 cents that this is something we already see and 
experience with hRaven so it's not theoretical.

Sangjin, thanks for sharing the use case in hRaven. It's helpful to understand 
the proper defaults. To generalize it, we need to consider different use cases 
such as adhoc applications only. Shall we continue the discussion on YARN-3461?

bq. As I mentioned earlier, it should be useful for developers

I make use of Sangjin's previous comments to add some inline code comments 
about their definitions in TimelineCollectorContext.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-07 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484512#comment-14484512
 ] 

Junping Du commented on YARN-3391:
--

bq. I make use of Sangjin's previous comments to add some inline code comments 
about their definitions in TimelineCollectorContext.
I would expect the definition can show up in Javadoc of related methods in 
TimelineCollectorContext. This sounds like a little nitpick, but the key 
differences between inline comments and javadoc is if developer only use jar 
instead of source code, they can still read these key definitions and use it 
correctly (by IDE hint or generated Javadoc). So in general, I think we should 
use as much javadoc comments instead of inline comments for public APIs. 

{code}
+// Sanity check for flow run
+try {
+  for (String tag : submissionContext.getApplicationTags()) {
+if (tag.startsWith(TimelineUtils.FLOW_RUN_TAG_PREFIX + :) ||
+tag.startsWith(
+TimelineUtils.FLOW_RUN_TAG_PREFIX.toLowerCase() + :)) {
+  String value =
+  tag.substring(TimelineUtils.FLOW_RUN_TAG_PREFIX.length() + 1);
+  Long.valueOf(value);
+}
+  }
+} catch (NumberFormatException e) {
+  LOG.warn(Invalid to flow run., e);
+  RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
+  e.getMessage(), ClientRMService,
+  Exception in submitting application, applicationId);
+  throw RPCUtil.getRemoteException(e);
+}
{cide}
We should add more info to LOG.warn messages, at least to tell user flow run 
should be numeric. In addition, do we need to check negative value for flow run 
here? If not, why we are accepting negative long value but rejecting other 
characters than number?

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-07 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484514#comment-14484514
 ] 

Junping Du commented on YARN-3391:
--

Sorry that the 2nd comment above has format issue and may hard to read. Fix the 
comments as below:
In ClientRMService.java, 
{code}
+// Sanity check for flow run
+try {
+  for (String tag : submissionContext.getApplicationTags()) {
+if (tag.startsWith(TimelineUtils.FLOW_RUN_TAG_PREFIX + :) ||
+tag.startsWith(
+TimelineUtils.FLOW_RUN_TAG_PREFIX.toLowerCase() + :)) {
+  String value =
+  tag.substring(TimelineUtils.FLOW_RUN_TAG_PREFIX.length() + 1);
+  Long.valueOf(value);
+}
+  }
+} catch (NumberFormatException e) {
+  LOG.warn(Invalid to flow run., e);
+  RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
+  e.getMessage(), ClientRMService,
+  Exception in submitting application, applicationId);
+  throw RPCUtil.getRemoteException(e);
+}
{code}
We should add more info to LOG.warn messages, at least to tell user flow run 
should be numeric. In addition, do we need to check negative value for flow run 
here? If not, why we are accepting negative long value but rejecting other 
characters than number?

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch, YARN-3391.2.patch, YARN-3391.3.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-02 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393264#comment-14393264
 ] 

Zhijie Shen commented on YARN-3391:
---

[~vrushalic], it sounds good to me to set aside the disagreement on the flow 
name default and move on. As far as I can tell, with the current context info 
data flow, it's quite simple to change the default value if we figure out the 
better one later. In addition, the previous debate is also related how we show 
flows on the web UI by default. I think we can go back to visit the defaults 
once we reaches the web UI work when we should have a better idea about it.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-02 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393645#comment-14393645
 ] 

Sangjin Lee commented on YARN-3391:
---

I am fine with tabling this discussion and revisiting it later in the interest 
of making progress.

I just wanted to add my 2 cents that this is something we already see and 
experience with hRaven so it's not theoretical. That's the context from our 
side. The way I see it is that apps that do not have the flow name are 
basically a degenerate case of a single-app flow. This is unrelated to the 
app-to-flow aggregation. It has to do with the flowRun-to-flow aggregation. And 
it's something we want the users to do when they can set the flow name. FWIW...

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-02 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393150#comment-14393150
 ] 

Vrushali C commented on YARN-3391:
--

Hi [~zjshen]

In the interest of time, I think let's park these disagreements aside and move 
forward with your defaults. If the need arises, we could revisit defaults in 
the future. What do you all think? cc [~sjlee0]  

thanks
Vrushali

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390900#comment-14390900
 ] 

Sangjin Lee commented on YARN-3391:
---

Hi [~djp],

The flow id identifies a distinct flow application that can be run repeatedly 
over time. The flow run id identifies one instance (or specific execution) of 
that flow. Finally, the flow version keeps track of the changes made to the 
flow (e.g. changes to the source code).

Let me give you a concrete example. Suppose you have a pig script you run 
repeatedly, named tracking.pig. The flow id in this case may be 
tracking.pig (or al...@tracking.pig to denote the fact that user alice 
runs this script).

The tracking.pig script will be run repeatedly many times. If I run it today, 
that specific run may have the flow run id of 1427846400 (timestamp when the 
pig script started). If I run it again tomorrow, the run id of that run would 
be 1427932800, and so on. Multiple run id's for the same flow id is a series 
of runs of the same script.

The flow version identifies changes made to the flow (user application). One 
scheme may be to use some kind of a hash of the pig script. Another scheme may 
be to use the git commit hash. Or some real versions if the user application 
has well-defined versions.

A flow run is *NOT* a subset of YARN apps run inside a flow. A flow is a 
template of runs if you will, and a flow run is an actual run instances of that 
flow. These are described in some detail in the original design doc in 
YARN-2928.

I hope this helps.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390844#comment-14390844
 ] 

Junping Du commented on YARN-3391:
--

Thanks [~zjshen] for delivering the patch! 
To be honest, I am getting more confused on these concepts from some discussion 
above:
From what I was understanding, flow is a group of applications that will get 
run (sequential or parallel) in a batch, and flow_run is one run branch for 
subset of flow applications (apps in flow_run only get run in sequence, 
however, different flow_runs under one flow could run in parallel). Does flow 
version sounds like a timestamp concept (from HBase prospective) which 
represent a specific run time for the flow?
Just quickly go through the attached patch, I didn't find answer there. I think 
we should document the concept/definition of flow, flow run and flow 
version clearly in Javadoc (web doc could be later when we finish the feature) 
which could help reviewer and developers to understand better. 

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390941#comment-14390941
 ] 

Zhijie Shen commented on YARN-3391:
---

I'll put some description in the javadoc somewhere, but I think eventually we 
need to describe it clearly in the documentation of YTS v2.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Junping Du (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390936#comment-14390936
 ] 

Junping Du commented on YARN-3391:
--

Thanks [~sjlee0] for reply quickly! That helps a lot. I initially thought flow 
run is a run instance (may from YARN-2928 design doc or somewhere) but get 
confused to something else when I saw flow version. Thanks for bringing me 
back. :)
Given other contributors could miss discussions here, I would suggest we add 
Javadoc to explain these somewhere, e.g. in TimelineCollectorContext.java.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391864#comment-14391864
 ] 

Zhijie Shen commented on YARN-3391:
---

Sangjin, thanks for your comments, too. According to your and Joep's comments, 
I can see the benefit to show application aggregation information by 
application (type). However, IMHO, it's orthogonal to flow definition. Isn't 
the straightforward approach to provide this feature via aggregating on 
application name/type dimension instead of let flow name = application name.

On the other side, flow should semantically stand for *workflow* (correct me if 
I'm wrong about flow concept), which contains a group of applications that work 
together to resolve a problem. Making flow name == application name changes the 
semantics That said, a flow of applications means the applications of the same 
type.

{quote}
 If a user is running TestDFSIO over and over, they should be recognized as 
different instances of the same thing.
{quote}

I guess the same thing you had in mind is not the same workflow, but the same 
application type, right? How about we decoupling the two concepts? One step 
back, when users set the flow explicitly, are they going to tell the 
application that you belong to workflow abc, or that you belong to job type 
xyz? I think it will be the former.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391959#comment-14391959
 ] 

Vrushali C commented on YARN-3391:
--


For default values, workflow = appname is much more user friendly and intuitive 
than workflow name = flow_number_number. 

Setting the flow name to flow_number_number per run will mean the UI will 
have a lengthy list of flow_number_number (similar to JT/RM). This will not be 
a step up from current JT / RM UI experience.


 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391711#comment-14391711
 ] 

Sangjin Lee commented on YARN-3391:
---

OK, just to clarify, we're talking about a case where one flow (run) is one 
YARN app. The only debate is whether the repeated runs of the (essentially) 
same YARN app should be grouped as different runs of the same flow, or all 
different flows altogether. In other words, *if it ran 100 times, should we 
have 100 flow runs of one flow, or 100 flows each of which has exactly one flow 
run?*

To me it seems a no brainer (thanks [~vrushalic] for reminding me) that we do 
want to group the runs of the same YARN app. If a user is running TestDFSIO 
over and over, they should be recognized as different instances of the same 
thing.

One mitigating factor is we would modify the mapreduce code to provide the flow 
name/id in case it's not set. Then the default behavior won't kick in for the 
most part. But I think it is important enough to group them and surface them as 
instances of the same flow.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391179#comment-14391179
 ] 

Vrushali C commented on YARN-3391:
--

bq. Otherwise, if we use the job name, for example, all the wordcout jobs will 
belong to one flow then by default.

Yes, that's exactly what they are. All wordcount jobs belong to the same flow 
wordcount and each run of the word count is a flow run, isn't it? 

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391296#comment-14391296
 ] 

Vrushali C commented on YARN-3391:
--

I have some semantic level comments.
1) bq.  public static String generateDefaultFlowIdBasedOnAppId(ApplicationId 
appId) {
return flow_ + appId.getClusterTimestamp() + _ + appId.getId();

would be nice to have this string as a static final somewhere. Also the 
separator defined as a static final string. 

2) I see that flowRun means flowRunId in this code now. I would actually keep 
it as flowRunId. Because an api call like getFlowRun() to me seems that it 
should return the flow run details, not just the flow run id.

3) Reposting an earlier reply since jira seems to align it earlier in the 
thread. bq. Otherwise, if we use the job name, for example, all the wordcout 
jobs will belong to one flow then by default.

Yes, that's exactly what they are. All wordcount jobs belong to the same flow 
wordcount by that user and each run of the word count is a flow run. In fact, 
they should not end up being separate flows. 



 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391549#comment-14391549
 ] 

Zhijie Shen commented on YARN-3391:
---

I have offline discussion with Vrushali. Here's some summary:

1. We agree that by default, each individual application should belongs to each 
individual flow run. 

2. While Vrushali thought different applications of the same name should belong 
to the same flow (name), I prefer each individual application should belong to 
different flow (name).

My opinion is that each individual application should be completely separated 
at different flow notation levels unless users specify name/version/run 
explicitly to minimize the interaction with other applications. For example, 
the aggregation about this application won't affect others and wont be affected 
by others.

And one technical problem about using application name is that it's N/A by 
default, unless users set it explicitly in the framework code. Similarly, the 
other field that we could choose for flow name is application type, which is 
YARN by default. Therefore, either using name or type will potentially result 
in most of users' applications in the flow (name) N/A/YARN.

However, the more essential question is if it makes sense to group the 
applications by application name/type by default at the flow (name) level, and 
if the flow-level aggregation info makes sense for this default grouping (e.g. 
all wordcount  jobs of zjshen). [~sjlee0] and [~vinodkv], any comments?




 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Joep Rottinghuis (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391554#comment-14391554
 ] 

Joep Rottinghuis commented on YARN-3391:


Whether the app_id needs to be part of the default flow name or not seems to 
boil down how we think about flows.
Let's say somebody runs the Sleep job, wordcount, TestDFSIO, or an application 
that doesn't use MapReduce (where we could default to the app name). For 
example if somebody runs a Spark app.

Then are we thinking on the future RM UI, would we show 1 line for each:
{noformat}
Sleep 50 runs cost x 
wordcount 12 runs cost y
TestDFSIO 10 runs cost z
{noformat}

Or would we show one line per run:
{noformat}
Sleep_...1 1 runs cost x/50 
Sleep_...2 1 runs cost x/50 
...
Sleep_...49 1 runs cost x/50 
Sleep_...50 1 runs cost x/50 

wordcount_...1 1 runs cost y/12
wordcount_...2 1 runs cost y/12
wordcount_...3 1 runs cost y/12
...
wordcount_...11 1 runs cost y/12
wordcount_...11 1 runs cost y/12
TestDFSIO_1 1 runs cost z/10
TestDFSIO_2 1 runs cost z/10
TestDFSIO_3 1 runs cost z/10
...
TestDFSIO_9 1 runs cost z/10
TestDFSIO_10 1 runs cost z/10
{noformat}

It would seem that we already have the UI with individual application ids, so 
users can already see each individual yarn app that way. We'd also be able to 
drill into the wordcount flow name and see 12 runs, each with their unique yarn 
app id.
Therefore it seems to me that adding the app_id to the flow_id by default does 
not add any value, but setting the flow_id to  the app name does add value. We 
don't want to map it to a static value as pointed out earlier (we'd see a huge 
number of runs for a single flow called 1 or something similar), but forcing 
every flow to be unique seems to overlap with what we already have with runs. 
We'd force each flow to be unique with only 1 run.


 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-04-01 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391617#comment-14391617
 ] 

Zhijie Shen commented on YARN-3391:
---

Thanks for your input, Joep!

bq. Therefore it seems to me that adding the app_id to the flow_id by default 
does not add any value,

Yeah, I agree it's not adding value by using the app_id, but IMHO, it also 
doesn't add problem. Backing to the aforementioned example, if Sleep_...1 -- 
Sleep_...40 is using application name as the flow name, and Sleep_...41 -- 
Sleep_...50 is set explicitly to be part of flow XYZ, I'll get something weird 
on web UI that Sleep 40 runs cost 4/5 x. It misleads users that there're 40 
sleep jobs instead of 50.

bq. Then are we thinking on the future RM UI, would we show 1 line for each:

Instead, showing this information sounds more like aggregating applications 
according to application name/type. We can do the aggregation at these 
dimensions, jus as aggregating them based on queue and so on.


 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: YARN-3391.1.patch


 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-03-30 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387491#comment-14387491
 ] 

Vrushali C commented on YARN-3391:
--

bq. I propose:
flow name: String: default(cluster_appId without app prefix)

AppId is some string like application_epoch_timestamp_some_number . I don't 
think using just the numerical part without the app_ prefix will be easy to 
relate to. Actually, what would be easy for the user to relate to is something 
like (in case of hadoop jobs) mapreduce.job.name param from the config. 
[~zjshen] do you know of any such config or context parameter that can be set 
so that we can pick up the flow name from there for all yarn applications?

bq. flow version: String: default(1)
default string of 1 is fine. 

bq. flow run: long: default(1)

Using a run id of 1 will mean everything will fall into this bucket if no one 
sets the run id. There needs to be a way to ensure the run id is set or if not, 
the default needs to be something variable like submit_time. Else we would have 
a poblem with having a default run id of 1. For example, if I run a sleep job 
10 times and I don't set the run id, then information of each run is 
overwritten (since all of them will have run id of 1).





 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-03-30 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387462#comment-14387462
 ] 

Zhijie Shen commented on YARN-3391:
---

bq.  Then we can apply default values for all (name = YARN app name, version = 
1, run id = app submit time)

I propose:

* flow name: String: default(cluster_appId without app prefix)
* flow version: String: default(1)
* flow run: long: default(1)

The rationale behind flow name default is that for one type of application 
(e.g., MR wordcount job), the app name will be the same. Using app name will 
group these apps together into one flow.

The rationale behind flow run default is that given an orphan app, we need 
always query the app report for the submission time to compose the identifier 
(cluster id, user id, flow name, flow version, flow run, app id, entity type, 
entity id).

bq. So one option in this case is to reject the submission if the flow id is 
set but the flow run id is not set, but there may be better ways of handling 
cases like that.

Using 1 for flow run by default shouldn't have the issue, and we don't need to 
reject the app for it.

I'm working on a patch, and will post it soon.


 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-03-30 Thread Zhijie Shen (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387515#comment-14387515
 ] 

Zhijie Shen commented on YARN-3391:
---

bq. do you know of any such config or context parameter that can be set so that 
we can pick up the flow name from there for all yarn applications?

I propose to use app id to generate flow name, such that every orphan app will 
be put in a unique default flow. Otherwise, if we use the job name, for 
example,  all the wordcout jobs will belong to one flow then by default.

bq. Using a run id of 1 will mean everything will fall into this bucket if no 
one sets the run id.

It won't, because each orphan app will have a unique default flow.

Compare to multiple runs under the same flow as the default, I think the more 
appropriate way is that each application should have a unique flow name. Under 
one flow, there's only one default version 1, and under the version, there's 
only one run 1.



 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-03-25 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381021#comment-14381021
 ] 

Sangjin Lee commented on YARN-3391:
---

The easiest case is if the client didn't set anything (e.g. a standalone MR 
app). Then we can apply default values for all (name = YARN app name, version = 
1, run id = app submit time).

One interesting case is if it is a real multi-app flow and it didn't set the 
run id. If a particular flow run had 3 YARN apps, and the flow id was set but 
the run id wasn't set, if we use the YARN app submit time, these 3 YARN apps 
would get different run id's as their app submit times would be different, and 
that's not what we want. The submit time of the flow is not readily apparent 
during the interaction between the flow client and YARN.

So one option in this case is to reject the submission if the flow id is set 
but the flow run id is not set, but there may be better ways of handling cases 
like that.

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-03-25 Thread Sangjin Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381022#comment-14381022
 ] 

Sangjin Lee commented on YARN-3391:
---

That's correct. But it would be good to give some thought to it to make sure 
what we do here would be able to accommodate nested flows with minimal 
changes...

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (YARN-3391) Clearly define flow ID/ flow run / flow version in API and storage

2015-03-25 Thread Vrushali C (JIRA)

[ 
https://issues.apache.org/jira/browse/YARN-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380942#comment-14380942
 ] 

Vrushali C commented on YARN-3391:
--

bq. How do we handle flow attributes in case of nested levels of flows?

I think in this version of ATS, we are not looking at nested flows? 

 Clearly define flow ID/ flow run / flow version in API and storage
 --

 Key: YARN-3391
 URL: https://issues.apache.org/jira/browse/YARN-3391
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 To continue the discussion in YARN-3040, let's figure out the best way to 
 describe the flow.
 Some key issues that we need to conclude on:
 - How do we include the flow version in the context so that it gets passed 
 into the collector and to the storage eventually?
 - Flow run id should be a number as opposed to a generic string?
 - Default behavior for the flow run id if it is missing (i.e. client did not 
 set it)
 - How do we handle flow attributes in case of nested levels of flows?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)