Michael, Thanks for the thoughts. I agree with you that we could solve some of these issues with some simple servlets in the container.
As for the WebSockets I think its a good idea to help update the client, however, we may have some browser issues with this. I know that one of my user bases doesn't have browser that support web sockets. Chris On Mon, Sep 30, 2013 at 3:39 PM, Michael Blumberg < [email protected]> wrote: > Hey Chris, > > Thanks for the response, that helped clear it up. I asked because I had a > thought regarding a potential solution but wasn't sure how some of the > missing functionality of v1 worked and if my solution was even viable. > > I was thinking that implementing a WebSocketServlet on the Jetty server > could: > > - Track changes to the Blur queries and store queries < 30 minutes old > (in memory or filesystem) > - Track changes in the zookeeper status > - Send updates to the client via websockets. > > The AngularJS client app would need to open a websocket connection to the > WebSocketServlet. > > I think you touched on something like this above when you said you could > "Add on demand checking in the Jetty server" to track the Zookeeper > ensemble status. > > Anyway, just a thought. > > Cheers, > Michael > > > On Mon, Sep 30, 2013 at 12:39 PM, Chris Rohr <[email protected]> wrote: > > > Michael, > > > > Thank you for your question. Hopefully this answers the bill: > > > > Of the features that I listed not being available the following were done > > by the Java application (back-end) piece in v1: > > > > - Zookeeper ensemble status > > This feature utilized the Java zookeeper libraries to connect and > check > > the status. I have not found a Javascript Library that can do this > > (outside of using NodeJS). There are only 2 ways we can do this going > > forward if this feature is indeed needed, 1. Add the zookeeper status to > > the Blur API or 2. Add on demand checking in the Jetty server (not sure > > what the load would be due to this) > > > > - Ability to monitor more than one Blur instance in the same tool > > This feature was possible because the Java application had its own > > configuration and allowed many instances to be setup all at once. Since > > this will be started (hopefully) with the start-all.sh and it will be > using > > the blur-site.config for its configuration, it will not make sense to > have > > this version contain many instances > > > > - HDFS information > > The Java application did all polling HDFS node and capacity status as > > well as metric gathering. The Rails application did have a thrift > > interface that allowed file browsing, but this I believe can all be used > in > > other tools. > > > > - Keeping track of more queries than the Blur cache of 2 minutes > > The Java application would keep 30 minutes of queries so that we > could > > go back and see if something bad had happened. If we are using Blur > > exclusively and not storing in a DB now, than we can't maintain the > queries > > for a longer period of time, with the exception of allowing the browser > to > > keep track until page reload. So basically you would get 2 minutes on > > reload and then it would add in new ones as you stayed on the page. > > > > Hope this helps, > > Chris > > > > > > On Mon, Sep 30, 2013 at 10:34 AM, Michael Blumberg < > > [email protected]> wrote: > > > > > Hi Chris, > > > > > > You mentioned that certain features would not be in v2 for a variety of > > > reasons one of which was a lack of a server polling component. Which > > > feature(s) in particular is/are affected by this and is the lack of > > support > > > on the server or client side? > > > > > > -Michael > > > > > > > > > On Sun, Sep 29, 2013 at 8:33 PM, Chris Rohr <[email protected]> > > wrote: > > > > > > > All, > > > > > > > > As I have begun to start the next version of the console I wanted to > > give > > > > everyone my proposal for what should be in it and get any feedback > that > > > you > > > > may have. I have created a new branch (new-blur-console) and will be > > > > pushing it up as soon as I can. Please let me know if you have any > > > > questions/opinions/suggestions about this proposal. If I don't hear > > from > > > > anyone, I will just assume it looks good and I will move forward. > > > > > > > > Thanks in advance, > > > > Chris > > > > > > > > *Blur Console v2 Design Proposal* > > > > > > > > *Purpose* > > > > > > > > This writeup will help to determine what direction the console > portion > > of > > > > Blur should go. There has always been a need for a push button, easy > > > setup > > > > solution so that dev-ops could get up and running with the console > > > faster. > > > > *NOTE: *This is purely a suggestion and all opinions and > > recommendations > > > > are welcome. > > > > > > > > > > > > *Server* > > > > > > > > Java program that runs an embedded Jetty server. The purpose of the > > > Jetty > > > > server would be to serve up any of the html pages and resources > needed > > > for > > > > the page as well as potentially any JSON based calls. > > > > > > > > > > > > *Interaction with Blur* > > > > > > > > The console would utilize the Javascript generated Thrift API of Blur > > to > > > > communicate and retrieve information from Blur. > > > > > > > > > > > > *Main Site Framework* > > > > > > > > We are looking to use AngularJS to provide an MVC framework in the > > > browser > > > > so that we will not have to rely on a front-end component and a > > back-end > > > > component to this system. This should allow this tool to be much > more > > > > lightweight. > > > > > > > > > > > > *Features* > > > > > > > > The main page of the site would be broken into tabs for major pieces > of > > > > functionality. The list below will outline what each tab would > > provide: > > > > > > > > > > > > > > > > - Dashboard > > > > - Controller Node Status (chart) > > > > - Shard Node Status by Cluster (chart) > > > > - Query Load by Cluster (chart) > > > > - Query Performance by Cluster (chart) > > > > - Number of Tables (chart) > > > > - Long Running Query Warnings > > > > - Tables > > > > - Listing of Tables by Cluster and by Status > > > > - For each table > > > > - Shard Layout > > > > - Schema > > > > - Term Lookup > > > > - Actions > > > > - Disable Table > > > > - Enable Table > > > > - Delete Table (with or without deleting underlying index in > > > HDFS) > > > > - Queries > > > > - List of Running Queries (last 2 minutes) > > > > - For each Query > > > > - Display base info about the query (query string, user, > > status) > > > > - Extra info about query (selector information) > > > > - Search (live search of Blur) > > > > > > > > > > > > In addition we can have some base user preferences that will be > stored > > in > > > > local storage or cookies in the browser. > > > > > > > > > > > > *Things missing from v1* > > > > > > > > The following items will not be in v2 either because they are not in > > the > > > > Blur API yet or because of the lack of a server polling component. > > These > > > > are all things that were nice to have but aren’t as critical. If any > > of > > > > these become critical we can figure out how to provide this > information > > > to > > > > the console. > > > > > > > > > > > > > > > > - Zookeeper ensemble status > > > > - Ability to monitor more than one Blur instance in the same tool > > > (each > > > > instance will have its own Console now) > > > > - All HDFS pieces (deferring to other purpose built tools like > Hue) > > > > - User auditing of actions like disabling or deleting a table (we > > > could > > > > do this with the server component) > > > > - Keeping track of more queries than the Blur cache of 2 minutes > (we > > > > could do something that until a page refresh we could keep track > of > > > > more). > > > > > > > > > > > > *Other Items to consider* > > > > > > > > One other big item to consider is how to configure the console. I’m > > > > thinking that we could just figure out how to tie into the blur-site > > > > config. > > > > > > > > > >
