Hey Chris, Thanks for the response, that helped clear it up. I asked because I had a thought regarding a potential solution but wasn't sure how some of the missing functionality of v1 worked and if my solution was even viable.
I was thinking that implementing a WebSocketServlet on the Jetty server could: - Track changes to the Blur queries and store queries < 30 minutes old (in memory or filesystem) - Track changes in the zookeeper status - Send updates to the client via websockets. The AngularJS client app would need to open a websocket connection to the WebSocketServlet. I think you touched on something like this above when you said you could "Add on demand checking in the Jetty server" to track the Zookeeper ensemble status. Anyway, just a thought. Cheers, Michael On Mon, Sep 30, 2013 at 12:39 PM, Chris Rohr <[email protected]> wrote: > Michael, > > Thank you for your question. Hopefully this answers the bill: > > Of the features that I listed not being available the following were done > by the Java application (back-end) piece in v1: > > - Zookeeper ensemble status > This feature utilized the Java zookeeper libraries to connect and check > the status. I have not found a Javascript Library that can do this > (outside of using NodeJS). There are only 2 ways we can do this going > forward if this feature is indeed needed, 1. Add the zookeeper status to > the Blur API or 2. Add on demand checking in the Jetty server (not sure > what the load would be due to this) > > - Ability to monitor more than one Blur instance in the same tool > This feature was possible because the Java application had its own > configuration and allowed many instances to be setup all at once. Since > this will be started (hopefully) with the start-all.sh and it will be using > the blur-site.config for its configuration, it will not make sense to have > this version contain many instances > > - HDFS information > The Java application did all polling HDFS node and capacity status as > well as metric gathering. The Rails application did have a thrift > interface that allowed file browsing, but this I believe can all be used in > other tools. > > - Keeping track of more queries than the Blur cache of 2 minutes > The Java application would keep 30 minutes of queries so that we could > go back and see if something bad had happened. If we are using Blur > exclusively and not storing in a DB now, than we can't maintain the queries > for a longer period of time, with the exception of allowing the browser to > keep track until page reload. So basically you would get 2 minutes on > reload and then it would add in new ones as you stayed on the page. > > Hope this helps, > Chris > > > On Mon, Sep 30, 2013 at 10:34 AM, Michael Blumberg < > [email protected]> wrote: > > > Hi Chris, > > > > You mentioned that certain features would not be in v2 for a variety of > > reasons one of which was a lack of a server polling component. Which > > feature(s) in particular is/are affected by this and is the lack of > support > > on the server or client side? > > > > -Michael > > > > > > On Sun, Sep 29, 2013 at 8:33 PM, Chris Rohr <[email protected]> > wrote: > > > > > All, > > > > > > As I have begun to start the next version of the console I wanted to > give > > > everyone my proposal for what should be in it and get any feedback that > > you > > > may have. I have created a new branch (new-blur-console) and will be > > > pushing it up as soon as I can. Please let me know if you have any > > > questions/opinions/suggestions about this proposal. If I don't hear > from > > > anyone, I will just assume it looks good and I will move forward. > > > > > > Thanks in advance, > > > Chris > > > > > > *Blur Console v2 Design Proposal* > > > > > > *Purpose* > > > > > > This writeup will help to determine what direction the console portion > of > > > Blur should go. There has always been a need for a push button, easy > > setup > > > solution so that dev-ops could get up and running with the console > > faster. > > > *NOTE: *This is purely a suggestion and all opinions and > recommendations > > > are welcome. > > > > > > > > > *Server* > > > > > > Java program that runs an embedded Jetty server. The purpose of the > > Jetty > > > server would be to serve up any of the html pages and resources needed > > for > > > the page as well as potentially any JSON based calls. > > > > > > > > > *Interaction with Blur* > > > > > > The console would utilize the Javascript generated Thrift API of Blur > to > > > communicate and retrieve information from Blur. > > > > > > > > > *Main Site Framework* > > > > > > We are looking to use AngularJS to provide an MVC framework in the > > browser > > > so that we will not have to rely on a front-end component and a > back-end > > > component to this system. This should allow this tool to be much more > > > lightweight. > > > > > > > > > *Features* > > > > > > The main page of the site would be broken into tabs for major pieces of > > > functionality. The list below will outline what each tab would > provide: > > > > > > > > > > > > - Dashboard > > > - Controller Node Status (chart) > > > - Shard Node Status by Cluster (chart) > > > - Query Load by Cluster (chart) > > > - Query Performance by Cluster (chart) > > > - Number of Tables (chart) > > > - Long Running Query Warnings > > > - Tables > > > - Listing of Tables by Cluster and by Status > > > - For each table > > > - Shard Layout > > > - Schema > > > - Term Lookup > > > - Actions > > > - Disable Table > > > - Enable Table > > > - Delete Table (with or without deleting underlying index in > > HDFS) > > > - Queries > > > - List of Running Queries (last 2 minutes) > > > - For each Query > > > - Display base info about the query (query string, user, > status) > > > - Extra info about query (selector information) > > > - Search (live search of Blur) > > > > > > > > > In addition we can have some base user preferences that will be stored > in > > > local storage or cookies in the browser. > > > > > > > > > *Things missing from v1* > > > > > > The following items will not be in v2 either because they are not in > the > > > Blur API yet or because of the lack of a server polling component. > These > > > are all things that were nice to have but aren’t as critical. If any > of > > > these become critical we can figure out how to provide this information > > to > > > the console. > > > > > > > > > > > > - Zookeeper ensemble status > > > - Ability to monitor more than one Blur instance in the same tool > > (each > > > instance will have its own Console now) > > > - All HDFS pieces (deferring to other purpose built tools like Hue) > > > - User auditing of actions like disabling or deleting a table (we > > could > > > do this with the server component) > > > - Keeping track of more queries than the Blur cache of 2 minutes (we > > > could do something that until a page refresh we could keep track of > > > more). > > > > > > > > > *Other Items to consider* > > > > > > One other big item to consider is how to configure the console. I’m > > > thinking that we could just figure out how to tie into the blur-site > > > config. > > > > > >
