Is suspect we haven't written down the architecture thoroughly enough. There's some description of it in two recent presentations
1. Slider http://www.slideshare.net/steve_l/slider-a 2. YARN services http://www.slideshare.net/steve_l/yarn-services Slider is an YARN service; everything in that presentation matters. Page 22 also shows what state we keep in the AM -more specifically, what is persisted to HDFS, what is rebuilt from information from the running containers when the AM restarts -and what is lost on an AM restart. - The Slider application master takes the configuration JSON files in HDFS, and works to keep the actual count of every component matching that requested in resources.json - It asks YARN for new containers when needed, releases them when surplus. It persists the placement to a rolehistory file (more specificlaly, it saves a new file when things change). When new containers are needed (including on application start), if a component is flagged as wanting or needing placement where it was before, the old locations are used in the YARN requests - It supports plugin "providers" which handle the actual work of configuring and running the nodes. Historically we had some Java-based ones for each application; now we only have the AgentProvider. We've left the split in just in case we ever want to change providers, something that may be one route to better docker support (see the relevant JIRA there) - The agent provider runs a little python agent in every requested container (one container per component instance), which reports to the agent provider over an SSL connection. - The Agent Provider tells the agent what to do, e.g. install a binary, start it. - The AM keeps running, keeping the number of actual containers equal to the desired one (if there is enough space in the cluster) - Incoming events (yarn container messages, IPC calls) are stuck into an event queue and asynchronously processed - There's also a scheduled event queue for delayed operations, these are promoted to the main event queue when their scheduled time is reached. - If the AM fails, the components keep running; - The AM is restarted, reloads its persistent state and brings up new web site/services on the new host and ports. It receives the list of containers from YARN, determines the component type in each of these containers, and rebuilds its state. - the agents in the containers look for the restarted AM's new agent web service via the YARN registry. - The Agent Provider gets those notifications and rebuilds its state. All containers running the actual target application keeps running. All containers not running the application are released and replacements requested. This is because we don't know why the app isn't running or what state it was through the install/launch process. It's easier to start from the beginning. - If the running program finishes, the agent finishes, which triggers a YARN notification. The AM notes the event, and if unexpected increments the failure count for that component type (RoleStatus class). - A new container is then requested and the agent install/run begins again. - The component failure count will trigger the slider app failure if it reaches a threshold, this is how slider knows to give up if an application just isn't working. There's a scheduled count resetter to to stop a few failures over a week triggering application failure; we care more about the number of failures per hour. - There's a very minimal Web UI and IPC channel for some operations, only two of which change the AM state: flex and stop. Flex hands the AM an updated resources.json, which the AM then persists and reacts to by changing the desired state of the cluster. - The CLI class, SliderClient can be used as an API -Ambari does this so that it can deploy an application via Slider and manage it. (There's a bit more to the Ambari integration; ask if you are curious) Ongoing work includes - A REST API to replace the IPC channel; see SLIDER-151. - Rolling upgrades of an application so you don't need to stop HBase to update it. (SLIDER-290) - Docker support (SLIDER-532) - better component placement (better anti-affinity for applications like Kafka) - helping get more packages up, which will drive what improvements we need in slider. Some of these changes are going to need YARN enhancements (REST, placement), so are going to only be usable in production on future Hadoop versions (Hadoop 2.8+). We're going to try to put off a Hadoop 2.8+ only switch as long as possible. That's pretty much it. On 11 December 2014 at 02:05, liujs3 <[email protected]> wrote: > Hi, my company use slider to run some application like spark and > hbase,they let me show the slider's architecture,and run it on our product > line . could you give some architecture's study or report? > Now every day I study the source ..it always looks like not clear and > personal. > > > > > > > Best Regards! > ocean floyd > 亚信科技(中 国)有限公司 > www.asiainfo.com > BDX > Mp:18513780630 > E-mail:[email protected] > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
