Repository: falcon Updated Branches: refs/heads/master 145f9bc77 -> f191ea540
FALCON-2078 Documentation for Process SLA Author: Praveen Adlakha <[email protected]> Reviewers: @sepreece, @pallavi-rao Closes #281 from PraveenAdlakha/2078 and squashes the following commits: 1dcf8e1 [Praveen Adlakha] comments addressed e37832e [Praveen Adlakha] comments addressed 94bdcc0 [Praveen Adlakha] Falcon-2078 Documentation for Process SLA Project: http://git-wip-us.apache.org/repos/asf/falcon/repo Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/f191ea54 Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/f191ea54 Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/f191ea54 Branch: refs/heads/master Commit: f191ea540e77fafa9b6e39aa9079e7dd375dfed8 Parents: 145f9bc Author: Praveen Adlakha <[email protected]> Authored: Wed Oct 19 16:30:50 2016 +0530 Committer: Pallavi Rao <[email protected]> Committed: Wed Oct 19 16:30:50 2016 +0530 ---------------------------------------------------------------------- .../falcon/persistence/EntitySLAAlertBean.java | 2 +- docs/src/site/twiki/EntitySLAAlerting.twiki | 57 +++++++++++++++ docs/src/site/twiki/EntitySLAMonitoring.twiki | 25 +++++++ docs/src/site/twiki/FalconDatabase.twiki | 74 ++++++++++++++++++++ docs/src/site/twiki/FalconDocumentation.twiki | 2 +- docs/src/site/twiki/FeedSLAMonitoring.twiki | 27 ------- .../site/twiki/GraphiteMetricCollection.twiki | 8 ++- docs/src/site/twiki/Operability.twiki | 5 +- 8 files changed, 167 insertions(+), 33 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java ---------------------------------------------------------------------- diff --git a/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java b/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java index 1419f48..186c5e0 100644 --- a/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java +++ b/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java @@ -35,7 +35,7 @@ import javax.validation.constraints.NotNull; //SUSPEND CHECKSTYLE CHECK LineLengthCheck /** - * Feed SLA monitoring. + * Entity SLA monitoring. * */ @Entity @NamedQueries({ http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/EntitySLAAlerting.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/EntitySLAAlerting.twiki b/docs/src/site/twiki/EntitySLAAlerting.twiki new file mode 100644 index 0000000..8534ba6 --- /dev/null +++ b/docs/src/site/twiki/EntitySLAAlerting.twiki @@ -0,0 +1,57 @@ +---++Entity SLA Alerting + +Falcon supports SLA in feed and process. + +Types of SLA supported for feed: + + 1.slaLow + 1.slaHigh + +To know more about feedSla look at [[EntitySpecification][Feed Specification]] + +Types of SLA supported for process: + + 1.shouldStartIn + 1.shouldEndIn + +To know more about processSla look at [[EntitySpecification][Process Specification]] + +Falcon Entity Alerting service do the following things: + + 1.Monitor instances of feed and process and send notifications to all the listeners attached to it. + 1.In case of feed it notifies when an *slaHigh* miss happens. slaLow is not supported. + 1.In case of process it notifies when an SLA miss for *shouldEndIn* happens. shouldStartIn is not supported. + +Entity SLA Alert service depends upon [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]] to know which process and feed instances are to be monitored. + +*How to attach listeners:* + +You can write custom listeners to do some action whenever a process or feed instance misses its SLA. +To attach listeners please add below property in startup.properties: + +<verbatim> + +*.entityAlert.listeners=org.apache.customPath.customListener + +</verbatim> + +Currently Falcon natively supports [[BacklogMetricEmitterService][Back Log Emitter Service]] as a listener to EntitySLAAlert service + +---++Dependencies: + +*Other Services:* + +To enable Enity SLA Alerting service you need to enable [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]] + + Following properties are needed in startup.properties: + +<verbatim> + +*.application.services=org.apache.falcon.service.EntitySLAAlertService + +*.entity.sla.statusCheck.frequency.seconds=600 +</verbatim> + +*Falcon Database:* + +Entity SLA Alerting service maintains its state in the database.It needs one table *ENTITY_SLA_ALERTS* please have a look at [[FalconDatabase]] to know how to create it. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/EntitySLAMonitoring.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/EntitySLAMonitoring.twiki b/docs/src/site/twiki/EntitySLAMonitoring.twiki new file mode 100644 index 0000000..bdd9ac4 --- /dev/null +++ b/docs/src/site/twiki/EntitySLAMonitoring.twiki @@ -0,0 +1,25 @@ +---++Falcon Entity SLA Monitoring + +Entity SLA monitoring allows you to monitor the entity (process and feed) .It keeps track of the instances of the entity that are running and stores them in the db. + + +---++Dependencies: + +*Other Services:* + +Entity SLA monitoring service requires FalconJPAService to be up. Following are the values you need to set to run EntitySLAMonitoring. +In startup.properties: + +<verbatim> +*.application.services= org.apache.falcon.state.store.service.FalconJPAService, + org.apache.falcon.service.EntitySLAMonitoringService +</verbatim> + + +*Falcon Database:* + +Entity SLA monitoring service maintains its state in the database.It needs two tables: + + 1.MONITORED_ENTITY + 1.PENDING_INSTANCES +please have a look at [[FalconDatabase]] to know how to create them. http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/FalconDatabase.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/FalconDatabase.twiki b/docs/src/site/twiki/FalconDatabase.twiki new file mode 100644 index 0000000..5800fa7 --- /dev/null +++ b/docs/src/site/twiki/FalconDatabase.twiki @@ -0,0 +1,74 @@ +---+++ Configuring the state store for Falcon +You can configure statestore by making changes to __$FALCON_HOME/conf/statestore.properties__ as follows. You will need to restart Falcon Server for the changes to take effect. + +Falcon Server needs to maintain state of the entities and instances in a persistent store for the system to be recoverable. Since Prism only federates, it does not need to maintain any state information. Following properties need to be set in statestore.properties of Falcon Servers: +<verbatim> +######### StateStore Properties ##### +*.falcon.state.store.impl=org.apache.falcon.state.store.jdbc.JDBCStateStore +*.falcon.statestore.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver +*.falcon.statestore.jdbc.url=jdbc:derby:data/falcon.db +# StateStore credentials file where username,password and other properties can be stored securely. +# Set this credentials file permission 400 ;the user who starts falcon should only have read permission. +# Give Absolute path to credentials file along with file name or put in classpath with file name statestore.credentials. +# Credentials file should be present either in given location or class path, otherwise falcon won't start. +*.falcon.statestore.credentials.file= +*.falcon.statestore.jdbc.username=sa +*.falcon.statestore.jdbc.password= +*.falcon.statestore.connection.data.source=org.apache.commons.dbcp.BasicDataSource +# Maximum number of active connections that can be allocated from this pool at the same time. +*.falcon.statestore.pool.max.active.conn=10 +*.falcon.statestore.connection.properties= +# Indicates the interval (in milliseconds) between eviction runs. +*.falcon.statestore.validate.db.connection.eviction.interval=300000 +## The number of objects to examine during each run of the idle object evictor thread. +*.falcon.statestore.validate.db.connection.eviction.num=10 +## Creates Falcon DB. +## If set to true, Falcon creates the DB schema if it does not exist. If the DB schema exists is a NOP. +## If set to false, Falcon does not create the DB schema. If the DB schema does not exist it fails start up. +*.falcon.statestore.create.db.schema=true +</verbatim> + +The _*.falcon.statestore.jdbc.url_ property in statestore.properties determines the DB and data location. All other properties are common across RDBMS. + +*NOTE : Although multiple Falcon Servers can share a DB (not applicable for Derby DB), it is recommended that you have different DBs for different Falcon Servers for better performance.* + +You will need to create the state DB and tables before starting the Falcon Server. To create tables, a tool comes bundled with the Falcon installation. You can use the _falcon-db.sh_ script to create tables in the DB. The script needs to be run only for Falcon Servers and can be run by any user that has execute permission on the script. The script picks up the DB connection details from __$FALCON_HOME/conf/statestore.properties__. Ensure that you have granted the right privileges to the user mentioned in statestore.properties_, so the tables can be created. + +You can use the help command to get details on the sub-commands supported: +<verbatim> +./bin/falcon-db.sh help +usage: + Falcon DB initialization tool currently supports Derby DB/ Mysql + + falcondb help : Display usage for all commands or specified command + + falcondb version : Show Falcon DB version information + + falcondb create <OPTIONS> : Create Falcon DB schema + -run Confirmation option regarding DB schema creation/upgrade + -sqlfile <arg> Generate SQL script instead of creating/upgrading the DB + schema + + falcondb upgrade <OPTIONS> : Upgrade Falcon DB schema + -run Confirmation option regarding DB schema creation/upgrade + -sqlfile <arg> Generate SQL script instead of creating/upgrading the DB + schema + +</verbatim> +Currently, MySQL, postgreSQL and Derby are supported as state stores. We may extend support to other DBs in the future. Falcon has been tested against MySQL v5.5 and PostgreSQL v9.5. If you are using MySQL ensure you also copy mysql-connector-java-<version>.jar under __$FALCON_HOME/server/webapp/falcon/WEB-INF/lib__ and __$FALCON_HOME/client/lib__ + +---++++ Using Derby as the State Store +Using Derby is ideal for QA and staging setup. Falcon comes bundled with a Derby connector and no explicit setup is required (although you can set it up) in terms creating the DB or tables. +For example, + <verbatim> *.falcon.statestore.jdbc.url=jdbc:derby:data/falcon.db;create=true </verbatim> + + tells Falcon to use the Derby JDBC connector, with data directory, $FALCON_HOME/data/ and DB name 'falcon'. If _create=true_ is specified, you will not need to create a DB up front; a database will be created if it does not exist. + +---++++ Using MySQL as the State Store +The jdbc.url property in statestore.properties determines the DB and data location. +For example, + <verbatim> *.falcon.statestore.jdbc.url=jdbc:mysql://localhost:3306/falcon </verbatim> + + tells Falcon to use the MySQL JDBC connector, which is accessible @localhost:3306, with DB name 'falcon'. + + Note: First time we have to manually create the schema in production as we have set falcon.statestore.create.db.schema = false \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/FalconDocumentation.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/FalconDocumentation.twiki b/docs/src/site/twiki/FalconDocumentation.twiki index fe1c0de..96d37a2 100644 --- a/docs/src/site/twiki/FalconDocumentation.twiki +++ b/docs/src/site/twiki/FalconDocumentation.twiki @@ -823,7 +823,7 @@ Extensions is detailed in [[Extensions][Extensions]]. ---++ Monitoring -Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]] and [[FeedSLAMonitoring][Falcon Feed SLA Monitoring]]. +Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]] and [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]]. ---++ Email Notification Notification for instance completion in Falcon is defined in [[FalconEmailNotification][Falcon Email Notification]]. http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/FeedSLAMonitoring.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/FeedSLAMonitoring.twiki b/docs/src/site/twiki/FeedSLAMonitoring.twiki deleted file mode 100644 index 469c0aa..0000000 --- a/docs/src/site/twiki/FeedSLAMonitoring.twiki +++ /dev/null @@ -1,27 +0,0 @@ ----++Falcon Feed SLA Monitoring - -Feed SLA monitoring allows you to monitor the feed ,It keeps track of the instances of the feed which are running and stores them in the db. - -Feed SLA monitoring service requires FalconJPAService to be up.Following are the values we need to set to run FeedSLAMonitoring - -In startup.properties : - -*.application.services= org.apache.falcon.state.store.service.FalconJPAService, - org.apache.falcon.service.EntitySLAMonitoringService - -These properties are required for FalconJPAService in statestore.properties: - - * *falcon.state.store.impl* - org.apache.falcon.state.store.jdbc.JDBCStateStore - * *falcon.statestore.jdbc.driver* -org.apache.derby.jdbc.EmbeddedDriver - * *falcon.statestore.jdbc.url* -jdbc:derby:target/test-data/data.db;create=true - * *falcon.statestore.connection.data.source* -org.apache.commons.dbcp.BasicDataSource - Maximum number of active connections that can be allocated from this pool at the same time. - * *falcon.statestore.pool.max.active.conn* -10 - * *falcon.statestore.connection.properties* - - Indicates the interval (in milliseconds) between eviction runs. - * *falcon.statestore.validate.db.connection.eviction.interval* -300000 - The number of objects to examine during each run of the idle object evictor thread. - * *falcon.statestore.validate.db.connection.eviction.num* -10 - Creates Falcon DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. - * *falcon.statestore.create.db.schema* -false - -Note: First time we have to manually create the schema in production as we have set falcon.statestore.create.db.schema = false \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/GraphiteMetricCollection.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/GraphiteMetricCollection.twiki b/docs/src/site/twiki/GraphiteMetricCollection.twiki index c76e68d..0ae0498 100644 --- a/docs/src/site/twiki/GraphiteMetricCollection.twiki +++ b/docs/src/site/twiki/GraphiteMetricCollection.twiki @@ -8,15 +8,17 @@ Graphite Metric Collection currently allows to collect the following metrics at To send data to graphite we need to intialize metricNotificationService in startup.properties: +<verbatim> *.application.services= org.apache.falcon.metrics.MetricNotificationService, - +</verbatim> Add following properties for graphiteNotificationPlugin : -Graphite properties +*Graphite properties* +<verbatim> * *.falcon.graphite.hostname=localhost * *.falcon.graphite.port=2003 * *.falcon.graphite.frequency=1 * *.falcon.graphite.prefix=falcon - +</verbatim> The falcon.graphite.frequency is in seconds and all the time that is being sent to graphite is in seconds. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/Operability.twiki ---------------------------------------------------------------------- diff --git a/docs/src/site/twiki/Operability.twiki b/docs/src/site/twiki/Operability.twiki index 2bccb51..f01c235 100644 --- a/docs/src/site/twiki/Operability.twiki +++ b/docs/src/site/twiki/Operability.twiki @@ -207,13 +207,16 @@ public class FalconUserJMSClient { ---++ Alerts -Falcon generates alerts for unrecoverable errors into a log file by default. +Falcon generates two type of alerts: + +1. By default it logs unrecoverable errors into a log file Users can view these alerts in the alerts.log file, by default this file is created under ${user.dir}/logs/ directory. Users may also extend the Falcon Alerting plugin to send events to systems like Nagios, etc. by extending org.apache.falcon.plugin.AlertingPlugin interface. +2. Alerts on *SLA misses* for feeds and process is detailed in [[EntitySLAAlerting][Entity SLA Alerting]]. ---++ Audits
