Zhijie Shen commented on YARN-1530:

[~sjlee0], thanks for your feedback. Here're some additional thoughts and 
clarifications upon your comments.

bq. This option would make sense only if the imports are less frequent.

To be more specific, I mean sending the same amount of entities (not too big, 
if too big HTTP REST request has to chunk them into some continuous HTTP 
requests with reasonable size) via HTTP REST or HDFS should perform similar. 
HTTP REST may be better because of less secondary storage I/O (ethernet should 
be fast than disk). HTTP REST doesn't prevent the user from batching the 
entities and put them once, and current API supports it. It's up to the user to 
put the entity immediately for realtime/near-realtime inquiry, or to batch 
entities if the can tolerant some delay.

However, I agree HDFS or some other single-node storage technique is a 
interesting part to prevent losing the entities when they are not published to 
the timeline server yet, in particular when we batching them.

bq. Regarding option (2), I think your point is valid that it would be a 
transition from a thin client to a fat client.
bq. However, I'm not too sure if it would make changing the data store much 
more complicated than other scenarios.

I'm also not very sure about the necessary changes. As what I mentioned before, 
timeline server doesn't simply put the entities into the data store. One 
immediate problem I can come up with is the authorization. I'm not sure it's 
going to be logically correct to check the user's access in the client at the 
user's side. If we move authorization to the data store, HBase supports access 
control, but Levedb seems not. And I'm not sure HBase access control is enough 
for timeline sever's specific logic. Still need to think more about it.

As the client is growing fatter, it's difficult to maintain different versions 
of clients. For example, if we do some incompatible optimization for the 
storage schema, only the new client can write into it, while the old client 
will not work any more. Moreover, as most writing logic is conducted at user 
land, which is not predictable, it is likely to raise some unexpected failure 
than a well setup server. In general, I prefer to keep the client simple, such 
that the future client distribution and maintenance could be of less effort.

bq. But then again, if we consider a scenario such as a cluster of ATS 
instances, the same problem exists there.

Right the same problem will exist at the server side, but the web front has 
isolated it from the users. Compared to the clients at the application, the ATS 
instances are a relatively small controllable set that we can pause them and do 
proper upgrading process. How do you think?

> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>                 Key: YARN-1530
>                 URL: https://issues.apache.org/jira/browse/YARN-1530
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>         Attachments: ATS-Write-Pipeline-Design-Proposal.pdf, 
> ATS-meet-up-8-28-2014-notes.pdf, application timeline design-20140108.pdf, 
> application timeline design-20140116.pdf, application timeline 
> design-20140130.pdf, application timeline design-20140210.pdf
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework 
> data all by itself as YARN doesn't have a common solution. This JIRA attempts 
> to solve the storage, management and serving of per-framework data from 
> various applications, both running and finished. The aim is to change YARN to 
> collect and store data in a generic manner with plugin points for frameworks 
> to do their own thing w.r.t interpretation and serving.

This message was sent by Atlassian JIRA

Reply via email to