Hi Anuruddha, Let's say the solution is already deployed. I need to add one more log agent to the deployment. For example I'm going to deploy an APIM instance and I need to send these logs, and I currently don't have access to add configurations to the master server.( I'm a new user)
To do this, according to how you've described above, that user has to have access to the master server. correct? If that's the case, then either all the users configuring them would have access to change the master or there has to be someone responsible for adding the necessary log configurations? Wouldn't that be cumbersome? Basically the questions I have are, 1. Why would you move the configurations to the master server? ( contrary to other popular implementations such as logstash) which makes the life easier in distributed setups. 2. Would every user who has access to the dashboard, be able to change the configurations as well in the master server ( permission model) ? On Mon, Oct 19, 2015 at 2:51 PM, Anuruddha Premalal <[email protected]> wrote: > Hi All, > > Here is a high level overview of the main components of the log analysis > tool. Appreciate your input on improving the use case. > > > *Log publishing agent* > > > > - > > These agents will connect to the analysis server (which can be > considered as the master) - agents will be installed on the host machine > with only a minimal configurations where it has information about the > master and, authentication information > > > - > > Master will list down the available agents, and will provide a wizard > to perform the configurations - Here master is the central point to > distribute configurations to the clients, this is similar to the puppet > approach of configuring clients, a group identification configuration can > be used in a clustered scenario where all the members of the grupId will > receive a similar set of configuration. > > > - > > Configurations wizard will walk the user through two main stages > > > > 1. Input configurations > - > > Initially this will be a file source, configurations related to > file input will be provided here. > - > > Moving forward this will be expanded to a logstash kind of approach > [2] , where users can define different kinds of sources. (Ex : syslog, > twitter, websockets, etc.. ) > > > > 1. Filtering configurations > - > > At this stage user can configure, how should the log events to be > filtered/ what sort of filters needs to be applied to the event. These > filters will apply a regex pattern/s in to the event and extract fields. > Pre-configured set of filters will be provided which can be applied to > generic log events like log4j, apache httpd. > - > > Also, user can search and pick a sample event from the previously > configured file input and extract formats by highlighting the fields and > defining regex patterns. > - > > Output format of the filtered event will be a json which contains > user defined fields along with a field of the raw event (see the sample > [1]). > > > - Single agent can have multiples of the above kind of configuration > (file{}, filter{}, output {}) in order to publish more than one source > events. > > > Search > > > - > > Search will primarily be based on Lucene queries. > - > > Results will be shown in a tabular view where users can drill down > further using the extracted fields. > - > > There will also be an option to apply time constraints to the search > query. > - > > Relative to current time - Ex : Last 12 hours, Last 5 days > - > > There will be a side panel which will have stats about the event > fields. > - > > Once a field is clicked user will get a pop up pane with the top > matches related to that field (field is basically a regex pattern), with > the frequency of occurrence both as a percentage and a count. > > > Dashboards and Reporting > > > - > > The popup described in the search section will also have a section for > Reporting and Dashboards based on metrics like; > > Average over time, Top values, Maximum over time, Minimum over time, Top > values, Top values by time, Rare values, etc… > > - > > There will be two types of dashboards, Lucene query based and real > time (Siddhi query based). Idea is to provide a unified language for both > the real time and historic searches as improvements. > - > > Users will be provided with the option to export reports & dashboards > in CSV, PDF, Excel formats. > > > Alerts > > - > > Both the real time and Lucene searches can be used to create alerts. > (save as alerts) > - > > Alerts can be configured based on three main categories > - > > Per-result alerts - Based on a continuous real-time > search (Siddhi). > - > > Scheduled alerts - Runs a search according to a schedule > that user specifies when creating the alert. (Periodic Lucene queries) > - > > Rolling window alerts - Based on a continuous real-time search > happens in a user defined time window (Siddhi) > > > > - > > Can have multiple trigger actions based on the following trigger > conditions > - > > Number of results > - > > Number of Hosts > - > > Custom search > > > > Trigger actions - > > - > > Send email / SMS / Phone call > - > > Web hook - POST to a user defined web service with the alert > payload. > - > > Run a custom script > > > [1] { > "message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET > /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\"; > \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 > Firefox/25.0\"", > "@timestamp" => "2013-12-11T08:01:45.000Z", > "@version" => "1", > "host" => "cadenza", > "clientip" => "127.0.0.1", > "ident" => "-", > "auth" => "-", > "timestamp" => "11/Dec/2013:00:01:45 -0800", > "verb" => "GET", > "request" => "/xampp/status.php", > "httpversion" => "1.1", > "response" => "200", > "bytes" => "3891", > "referrer" => "\"http://cadenza/xampp/navi.php\"";, > "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; > rv:25.0) Gecko/20100101 Firefox/25.0\"" > } > > [2] https://www.elastic.co/guide/en/logstash/current/config-examples.html > > > > Regards, > -- > *Anuruddha Premalal* > Software Eng. | WSO2 Inc. > Mobile : +94717213122 > Web site : www.anuruddha.org > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sachith Withana Software Engineer; WSO2 Inc.; http://wso2.com E-mail: sachith AT wso2.com M: +94715518127 Linked-In: <http://goog_416592669>https://lk.linkedin.com/in/sachithwithana
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
