[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329744#comment-14329744
]
Sangjin Lee commented on YARN-3166:
---
{quote}
1. In TimelineClient, we keep the existing methods that operate on the old data
model, but mark them deprecated individually.
2. In TimelineClient, we create the new methods that operate on the new data
model.
{quote}
Just to note the obvious, that would not work if it is a public interface that
other code implements. If it is internal, yes, we could evolve the interface
that way, but it is a public interface, that is not an option... So for
TimelineClient specifically, I think it is true. But as a general rule, I think
there could be issues...
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329848#comment-14329848
]
Zhijie Shen commented on YARN-3166:
---
bq. But as a general rule, I think there could be issues...
Yeah, could be, but hopefully it won't be significant. in YARN-3240, I've made
patch to add new client APIs into TimelineClient. Taking a close look at
TimelineClientImpl, the code of operating on old data model is relative a small
piece, while will need to carry over most skeleton code as well as part of the
APIs, i.e. DT operations.
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329598#comment-14329598
]
Zhijie Shen commented on YARN-3166:
---
bq. In such a scenario if we try to Reuse REST then we might end up with lot of
If else...
I'm not sure if I understand your question correction, but I think we may not
end up with lot of if/else. The proposal is:
1. In TimelineClient, we keep the existing methods that operate on the old data
model, but mark them deprecated individually.
2. In TimelineClient, we create the new methods that operate on the new data
model.
The methods of operating new and old data models are separated.
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329393#comment-14329393
]
Naganarasimha G R commented on YARN-3166:
-
Hi [~zjshen] [~sjlee0]
bq. I'm not aware that RM has something similar to NM aux service to decouple
RM and aggregator, too.
+1 for this approach, even i was thinking in the same lines and actually i had
raised jira for AUX support in RM for some other functionality in YARN - 2267.
bq. The benefit is that we can reuse the whole skeleton, including http rest
wrapper, security and retry code, but just need to handle some different data
objects, and direct the request to a different location. Instead of deprecate
the whole class, we can deprecate the individual methods. Thought?
One problem i can think of, as per Sangjin's comments still we remove ATSV1 we
need to support both to work based on configurations enable either of one. In
such a scenario if we try to Reuse REST then we might end up with lot of If
else...
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329442#comment-14329442
]
Li Lu commented on YARN-3166:
-
Hello guys, sorry for the late reply...
bq. RM and NM modules will depend on timeline service module?
I agree that for RM, this is almost unavoidable. I think it's reasonable to let
RM/NMs (in future) depend on timeline services, but we need to be careful on
cyclic dependencies?
bq. What is the difference between TimelineStorage and TimelineStorageImpl?
We may need some renaming here. In my original thought, TimelineStorage class
translates operations based on our object model to data storage layer method
calls. These methods should be implemented by a TimelineStorageImpl object (and
its subclasses, of course).
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327987#comment-14327987
]
Sangjin Lee commented on YARN-3166:
---
[~gtCarrera9], sorry for my delayed response.
It looks good mostly. I have some feedback:
- As [~zjshen] mentioned, we need to sort out the RM/NM dependency on the
timelineservice module. The NM dependency is more of a fluke, but we need to
think about the RM dependency because it needs to start its own aggregator
service. I believe [~Naganarasimha] mentioned this in another JIRA. Perhaps
this is unavoidable if RM is going to start the aggregator? I am not aware of
any clean pluggable service mechanism for RM (like the aux services for NM).
Another idea if we don't want that is to move the base aggregator class into
yarn-server-common.
- I think as a rule, it would be good to make sure not to disturb the old ATS
classes. IIUC we're deprecating the old ATS classes, but we're not going to
modify them in an incompatible way (e.g. moving classes, removing classes,
changing interfaces, etc.), as that would be extremely disruptive once this is
merged.
- What is the difference between TimelineStorage and TimelineStorageImpl?
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14328344#comment-14328344
]
Zhijie Shen commented on YARN-3166:
---
1. If YARN-3087 can be resolved, we can probably make NM avoid the dependency
on the timeline service module.
2. I'm not aware that RM has something similar to NM aux service to decouple RM
and aggregator, too. It seems to be more elegant to have base-aggregator or
server-oriented interfaces in server-common, but I okay with both. In addition
to packaging, coupling with RM means that we cannot deploy RM aggregator with
different versions of RM.
3. We can add the method to operate the new data model into the existing client
class, instead of creating a new one. The benefit is that we can reuse the
whole skeleton, including http rest wrapper, security and retry code, but just
need to handle some different data objects, and direct the request to a
different location. Instead of deprecate the whole class, we can deprecate the
individual methods. Thought?
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326538#comment-14326538
]
Li Lu commented on YARN-3166:
-
Hi [~sjlee0], [~rkanter], [~zjshen] and [~vinodkv], would (anyone of) you mind
to take a look at the conclusion here? I'm trying to finalize our first draft
for module/package structures for timeline v2 here. Please feel free to let me
know if you have any concerns. Thanks!
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326987#comment-14326987
]
Zhijie Shen commented on YARN-3166:
---
bq. I assume they will post data through our clients, am I right here?
RM and NM should have the code to start the aggregator too.
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14326812#comment-14326812
]
Zhijie Shen commented on YARN-3166:
---
Thanks for raising the package structure definition. Some comments:
1. Should we sort out the package for RM and NM code of writing actual system
data through the aggregator?
2. RM and NM modules will depend on timeline service module?
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325231#comment-14325231
]
Li Lu commented on YARN-3166:
-
Some notes about the current code organization arrangements:
# *API records*: classes should be consistent with YARN-3041.
# *Client*: Currently we would like to extend the existing timeline service
client. We’re planning to extend the API of the current client by adding more
support to our v2 object model. We may also mark existing methods in timeline
client as deprecated. In phase 1, we are not planning to add async client
support.
# *Storage*: TimelineStorage should contain all APIs for the DAO layer. A
TimelineStorageImpl implements required methods used by TimelineStorage. (Same
as the current storage layer. )
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325243#comment-14325243
]
Li Lu commented on YARN-3166:
-
Comments are more than welcome. Specifically, we may want to decide the
arrangement on timeline aggregator first, since we may need to refactor part of
the code.
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14325229#comment-14325229
]
Li Lu commented on YARN-3166:
-
I've updated the proposed package/module structures. We may want to refactor
some of our existing aggregator code, mainly changing their names. Here are the
latest version:
h4. Module: hadoop-yarn-api
* package: {{o.a.h.yarn.api.records.timelineservice}}
** ApplicationEntity, ApplicationAttemptEntity, ClusterEntity, ContainerEntity,
FlowEntity, FlowRunEntity, HierarchicalTimelineEntity, TimelineEntity
** TimelineEvent
** TimelineMetric
** TimelineQueue
** TimelineUser
h4. Module: hadoop-yarn-client
* package: {{o.a.h.yarn.client.api}}
** TimelineClient
* package: {{o.a.h.yarn.client.api.impl}}
** TimelineClientImpl
h4. Module: hadoop-yarn-server/hadoop-yarn-server-timelineservice
* package: {{o.a.h.yarn.server.timelineservice.aggregator}}
** BaseAggregatorService - BaseTimelineAggregator
** AppLevelAggregatorService - AppLevelTimelineAggregator
** AppLevelServiceManager - TimelineAggregatorsCollection
** AppLevelServiceProvider - (TODO: Remove or rename)
** PerNodeAggregatorServer - PerNodeTimelineAggregatorsAuxService
** PerNodeAggregatorWebService - TimelineAggregatorWebService
* package: {{o.a.h.yarn.server.timelineservice.reader}}
** TimelineReader
** TimelineReaderPool
* package: {{o.a.h.yarn.server.timelineservice.storage}}
** TimelineStorage
** TimelineStorageImpl
** HBaseTimelineStorageImpl
** PhoenixTimelineStorageImpl
** InMemTimelineStorageImpl
h4. Under hadoop-yarn-common
For common YARN libraries. Modules/libraries added into this directory need to
be decided on a case-by-case basis.
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323697#comment-14323697
]
Zhijie Shen commented on YARN-3166:
---
Records is better to be in
{{org.apache.hadoop.yarn.api.records.timelineservice.*}}?
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321090#comment-14321090
]
Li Lu commented on YARN-3166:
-
Sent the message by mistake before I finish. Here's the rest part:
*Under hadoop-yarn-client*
{{o.a.h.yarn.client.api}}: TimelineClient for ATS v1, moved from
o.a.h.yarn.client.api (common). Consistent with Yarn/NM/AHS/AMRM clients. We
may want to mark this class as deprecated.
* TimelineClient
{{o.a.h.yarn.client.api.impl: TimelineClientImpl}} for ATS v1, moved from
{{o.a.h.yarn.client.api.impl}} (in common). Consistent with Yarn/NM/AHS/AMRM
clients.
* TimelineClientImpl
Note that currently we do not have plan to extend the v1 client to have async
APIs.
*Under hadoop-yarn-server*
{{o.a.h.yarn.server.timelineservice.aggregator}}: timeline service v2
aggregator implementation (already checked in)
* BaseAggregatorService
* AppLevelAggregatorService
* AppLevelServiceManager (Singleton)
* AppLevelServiceProvider
* PerNodeAggregatorServer
* PerNodeAggregatorWebService
We may further refactor these classes, mainly some name changes. However, the
basic structure is well-built here.
{{o.a.h.yarn.server.timelineservice.reader}}: timeline service v2 reader
implementation
* Reader
* ReaderManager (Singleton)
* ReaderManagerProvider
{{o.a.h.yarn.server.timelineservice.storage}}: basic storage layer
implementations for both aggregators and readers
TimelineStorage: storage adapter for aggregators/readers, load storage impls
dynamically
* StorageImpl: shared interface for all timeline storages
* HBaseStorageImpl
* PhoenixStorageImpl
* InMemStorageImpl
Pluggable, so more storage implementations are possible. Each storage implement
may create its own package to further organize itself.
*Under hadoop-yarn-common*
For common YARN libraries. Modules/libraries added into this directory need to
be decided on a case-by-case basis.
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321082#comment-14321082
]
Li Lu commented on YARN-3166:
-
Discussed with [~vinodkv], [~zjshen] and [~djp] offline, and here's a initial
plan for the structure of related modules/packages. This plan is designed on
top of [~sjlee0]'s original discussion in YARN-2928. Please feel free to
provide us feedback on this plan. Thanks!
General organization is based on the discussion under YARN-2928:
{code}
api: org.apache.hadoop.yarn.api.timelineservice.*
common: org.apache.hadoop.yarn.timelineservice.*
client: org.apache.hadoop.yarn.client.api.timelineservice.*
server: org.apache.hadoop.yarn.server.timelineservice.*
{code}
Under hadoop-yarn-api
{{o.a.h.yarn.api.records.timelineservice}}: Object model for timeline service
v2 (subject to change)
* Entity
** Workflow
** Run
** Application
* Metric
* Event
* (Maybe changing, but should be consistent with YARN-3041)
{{o.a.h.yarn.api.timelineservice}}: Read/Write api for timeline service v2
* TimelineServiceRead
* TimelineServiceWrite
[Source organization] Decide detailed package structures for timeline service
v2 components
---
Key: YARN-3166
URL: https://issues.apache.org/jira/browse/YARN-3166
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Li Lu
Assignee: Li Lu
Open this JIRA to track all discussions on detailed package structures for
timeline services v2. This JIRA is for discussion only.
For our current timeline service v2 design, aggregator (previously called
writer) implementation is in hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
want to put reader related implementations into hadoop-yarn-server's:
{{org.apache.hadoop.yarn.server.timelineservice.reader}}
Both readers and aggregators will expose features that may be used by YARN
and other 3rd party components, such as aggregator/reader APIs. For those
features, maybe we would like to expose their interfaces to
hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
