[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user asfgit closed the pull request at: https://github.com/apache/incubator-fluo-website/pull/23 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278587 --- Diff: docs/fluo/1.0.0-incubating/install.md --- @@ -0,0 +1,210 @@ +--- +layout: fluo-doc +title: Fluo Install Instructions +version: 1.0.0-incubating +--- + +Instructions for installing Apache Fluo and starting a Fluo application in YARN on a cluster where +Accumulo, Hadoop & Zookeeper are running. If you need help setting up these dependencies, see the +[related projects page][related] for external projects that may help. + +## Requirements + +Before you install Fluo, the following software must be installed and running on your local machine +or cluster: + +| Software| Recommended Version | Minimum Version | +|-|-|-| +| [Accumulo] | 1.7.2 | 1.6.1 | +| [Hadoop]| 2.7.2 | 2.6.0 | +| [Zookeeper] | 3.4.8 | | +| [Java] | JDK 8 | JDK 8 | + +## Obtain a distribution + +Before you can install Fluo, you will need to obtain a distribution tarball. It is recommended that +you download the [latest release][release]. You can also build a distribution from the master +branch by following these steps which create a tarball in `modules/distribution/target`: + +git clone https://github.com/apache/incubator-fluo.git +cd fluo/ +mvn package + +## Install Fluo + +After you obtain a Fluo distribution tarball, follow these steps to install Fluo. + +1. Choose a directory with plenty of space and untar the distribution: + +tar -xvzf fluo-1.0.0-incubating-bin.tar.gz + +2. Copy the example configuration to the base of your configuration directory to create the default +configuration for your Fluo install: + +cp conf/examples/* conf/ + +The default configuration will be used as the base configuration for each new application. + +3. Modify [fluo.properties] for your environment. However, you should not configure any +application settings (like observers). + +NOTE - All properties that have a default are set with it. Uncomment a property if you want +to use a value different than the default. Properties that are unset and uncommented must be +set by the user. + +4. Fluo needs to build its classpath using jars from the versions of Hadoop, Accumulo, and +Zookeeper that you are using. Choose one of the two ways below to make these jars available +to Fluo: + +* Set `HADOOP_PREFIX`, `ACCUMULO_HOME`, and `ZOOKEEPER_HOME` in your environment or configure +these variables in [fluo-env.sh]. Fluo will look in these locations for jars. +* Run `./lib/fetch.sh ahz` to download Hadoop, Accumulo, and Zookeeper jars to `lib/ahz` and +configure [fluo-env.sh] to look in this directory. By default, this command will download the +default versions set in [lib/ahz/pom.xml]. If you are not using the default versions, you can +override them: + +./lib/fetch.sh ahz -Daccumulo.version=1.7.2 -Dhadoop.version=2.7.2 -Dzookeeper.version=3.4.8 + +5. Fluo needs more dependencies than what is available from Hadoop, Accumulo, and Zookeeper. These + extra dependencies need to be downloaded to `lib/` using the command below: + +./lib/fetch.sh extra + +You are now ready to use the Fluo command script. + +## Fluo command script + +The Fluo command script is located at `bin/fluo` of your Fluo installation. All Fluo commands are +invoked by this script. + +Modify and add the following to your `~/.bashrc` if you want to be able to execute the fluo script +from any directory: + +export PATH=/path/to/fluo-1.0.0-incubating/bin:$PATH + +Source your `.bashrc` for the changes to take effect and test the script + +source ~/.bashrc +fluo + +Running the script without any arguments prints a description of all commands. + +./bin/fluo + +## Configure a Fluo application + +You are now ready to configure a Fluo application. Use the command below to create the +configuration necessary for a new application. Feel free to pick a different name (other than +`myapp`) for your application: + +fluo new myapp + +This command will create a directory for your application at `apps/myapp` of your Fluo install which +will contain a `conf` and `lib`. + +The `apps/myapp/conf` directory contains a copy of the `fluo.properties` from your default +configuration. This should be configured for your application: + +
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278509 --- Diff: _posts/blog/2016-09-27-1.0.0-incubating-is-released.md --- @@ -0,0 +1,12 @@ +--- +title: "Fluo 1.0.0-incubating is released!" +date: 2016-09-27 15:30:00 +--- + +Apache Fluo is happy to announce a [1.0.0-incubating][rel] release. This is the first +release of Fluo as an Apache project and the fourth release for the project. Starting +with this release, Fluo will follow [semver][semver] for all future API changes. See +the [release page][rel] for more information. --- End diff -- This was removed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278537 --- Diff: _posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md --- @@ -1,16 +1,19 @@ --- title: Fluo 1.0.0-incubating Release Notes -date: 2016-10-04 15:30:00 -version: 1.0.0-incubating +date: 2016-09-27 15:30:00 +version: fluo-1.0.0-incubating --- Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth -release for the project. Since the beta-2 release there were [167 commits][commits] by -[cjmctague], [ctubbsii], [Garvit244], [keith-turner], [mikewalch]. Information about all --- End diff -- This was removed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user mikewalch commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83278560 --- Diff: docs/fluo/1.0.0-incubating/metrics.md --- @@ -0,0 +1,117 @@ +--- +layout: fluo-doc +title: Fluo Metrics +version: 1.0.0-incubating +--- + +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report some 'default' metrics about an application that help users monitor its +performance. Users can also write code to report 'application-specific' metrics from their +applications. Both 'application-specific' and 'default' metrics share the same reporter configured +by [fluo.properties] and are described in detail below. + +## Configuring reporters + +Fluo metrics are not published by default. To publish metrics, configure a reporter in the 'metrics' +section of [fluo.properties]. There are several different reporter types (i.e Console, CSV, +Graphite, JMX, SLF4J) that are implemented using [Dropwizard]. The choice of which reporter to use +depends on the visualization tool used. If you are not currently using a visualization tool, there +is [documentation][grafana] for reporting Fluo metrics to Grafana/InfluxDB. + +## Metrics names + +When Fluo metrics are reported, they are published using a naming scheme that encodes additional +information. This additional information is represented using all caps variables (i.e `METRIC`) +below. + +Default metrics start with `fluo.class` or `fluo.system` and have following naming schemes: + +fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS +fluo.system.APPLICATION.REPORTER_ID.METRIC + +Application metrics start with `fluo.app` and have following scheme: + +fluo.app.REPORTER_ID.METRIC + +The variables below describe the additional information that is encoded in metrics names. + +1. `APPLICATION` - Fluo application name +2. `REPORTER_ID` - Unique ID of the Fluo oracle, worker, or client that is reporting the metric. +When running in YARN, this ID is of the format `worker-INSTANCE_ID` or `oracle-INSTANCE_ID` +where `INSTANCE_ID` corresponds to instance number. When not running in YARN, this ID consists +of a hostname and a base36 long that is unique across all fluo processes. +3. `METRIC` - Name of the metric. For 'default' metrics, this is set by Fluo. For 'application' +metrics, this is set by user. Name should be unique and avoid using period '.' in name. +4. `CLASS` - Name of Fluo observer or loader class that produced metric. This allows things like +transaction collisions to be tracked per class. + +## Application-specific metrics + +Application metrics are implemented by retrieving a [MetricsReporter] from an [Observer], [Loader], +or [FluoClient]. These metrics are named using the format `fluo.app.REPORTER_ID.METRIC`. + +## Default metrics + +Default metrics report for a particular Observer/Loader class or system-wide. + +Below are metrics that are reported from each Observer/Loader class that is configured in a Fluo +application. These metrics are reported after each transaction and named using the format +`fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS`. + +* tx_lock_wait_time - [Timer] +- Time transaction spent waiting on locks held by other transactions. +- Only updated for transactions that have non-zero lock time. +* tx_execution_time - [Timer] +- Time transaction took to execute. +- Updated for failed and successful transactions. +- This does not include commit time, only the time from start until commit is called. +* tx_with_collision - [Meter] +- Rate of transactions with collisions. +* tx_collisions - [Meter] +- Rate of collisions. +* tx_entries_set - [Meter] +- Rate of row/columns set by transaction +* tx_entries_read - [Meter] +- Rate of row/columns read by transaction that existed. +- There is currently no count of all reads (including non-existent data) +* tx_locks_timedout - [Meter] +- Rate of timedout locks rolled back by transaction. +- These are locks that are held for very long periods by another transaction that appears to be + alive based on zookeeper. +* tx_locks_dead - [Meter] +- Rate of dead locks rolled by a transaction. +- These are locks held by a process that appears to be dead according to zookeeper. +* tx_status_`` - [Meter] +- Rate of different ways (i.e ``) a transaction can terminate + +Below are system-wide metrics that are reported for the entire Fluo application. These metrics are +named using the format
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user ctubbsii commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83275841 --- Diff: _layouts/default.html --- @@ -31,8 +31,10 @@ -Documentation -Download +Releases --- End diff -- I don't think a link directly to the latest download would be appropriate unless it is accompanied by instructions on verifying the release and downloading hashes/sigs. We can do this on the releases posts, but an independent link directly to the download is more problematic. Maybe a link directly to the "Latest Release" instead? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user keith-turner commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83218320 --- Diff: _layouts/default.html --- @@ -31,8 +31,10 @@ -Documentation -Download +Releases --- End diff -- This is feedback based on my first impression. At first I was not sure where to download from the main page. I was thinking this could be called downloads instead of releases. My thinking is that users would look for download more frequently than release notes.I'm ok with leaving it as releases, I'm used to it now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user keith-turner commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83222182 --- Diff: docs/fluo/1.0.0-incubating/metrics.md --- @@ -0,0 +1,117 @@ +--- +layout: fluo-doc +title: Fluo Metrics +version: 1.0.0-incubating +--- + +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report some 'default' metrics about an application that help users monitor its +performance. Users can also write code to report 'application-specific' metrics from their +applications. Both 'application-specific' and 'default' metrics share the same reporter configured +by [fluo.properties] and are described in detail below. + +## Configuring reporters + +Fluo metrics are not published by default. To publish metrics, configure a reporter in the 'metrics' +section of [fluo.properties]. There are several different reporter types (i.e Console, CSV, +Graphite, JMX, SLF4J) that are implemented using [Dropwizard]. The choice of which reporter to use +depends on the visualization tool used. If you are not currently using a visualization tool, there +is [documentation][grafana] for reporting Fluo metrics to Grafana/InfluxDB. + +## Metrics names + +When Fluo metrics are reported, they are published using a naming scheme that encodes additional +information. This additional information is represented using all caps variables (i.e `METRIC`) +below. + +Default metrics start with `fluo.class` or `fluo.system` and have following naming schemes: + +fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS +fluo.system.APPLICATION.REPORTER_ID.METRIC + +Application metrics start with `fluo.app` and have following scheme: + +fluo.app.REPORTER_ID.METRIC + +The variables below describe the additional information that is encoded in metrics names. + +1. `APPLICATION` - Fluo application name +2. `REPORTER_ID` - Unique ID of the Fluo oracle, worker, or client that is reporting the metric. +When running in YARN, this ID is of the format `worker-INSTANCE_ID` or `oracle-INSTANCE_ID` +where `INSTANCE_ID` corresponds to instance number. When not running in YARN, this ID consists +of a hostname and a base36 long that is unique across all fluo processes. +3. `METRIC` - Name of the metric. For 'default' metrics, this is set by Fluo. For 'application' +metrics, this is set by user. Name should be unique and avoid using period '.' in name. +4. `CLASS` - Name of Fluo observer or loader class that produced metric. This allows things like +transaction collisions to be tracked per class. + +## Application-specific metrics + +Application metrics are implemented by retrieving a [MetricsReporter] from an [Observer], [Loader], +or [FluoClient]. These metrics are named using the format `fluo.app.REPORTER_ID.METRIC`. + +## Default metrics + +Default metrics report for a particular Observer/Loader class or system-wide. + +Below are metrics that are reported from each Observer/Loader class that is configured in a Fluo +application. These metrics are reported after each transaction and named using the format +`fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS`. + +* tx_lock_wait_time - [Timer] +- Time transaction spent waiting on locks held by other transactions. +- Only updated for transactions that have non-zero lock time. +* tx_execution_time - [Timer] +- Time transaction took to execute. +- Updated for failed and successful transactions. +- This does not include commit time, only the time from start until commit is called. +* tx_with_collision - [Meter] +- Rate of transactions with collisions. +* tx_collisions - [Meter] +- Rate of collisions. +* tx_entries_set - [Meter] +- Rate of row/columns set by transaction +* tx_entries_read - [Meter] +- Rate of row/columns read by transaction that existed. +- There is currently no count of all reads (including non-existent data) +* tx_locks_timedout - [Meter] +- Rate of timedout locks rolled back by transaction. +- These are locks that are held for very long periods by another transaction that appears to be + alive based on zookeeper. +* tx_locks_dead - [Meter] +- Rate of dead locks rolled by a transaction. +- These are locks held by a process that appears to be dead according to zookeeper. +* tx_status_`` - [Meter] +- Rate of different ways (i.e ``) a transaction can terminate + +Below are system-wide metrics that are reported for the entire Fluo application. These metrics are +named using the format
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user keith-turner commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83219860 --- Diff: _posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md --- @@ -1,16 +1,19 @@ --- title: Fluo 1.0.0-incubating Release Notes -date: 2016-10-04 15:30:00 -version: 1.0.0-incubating +date: 2016-09-27 15:30:00 +version: fluo-1.0.0-incubating --- Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth -release for the project. Since the beta-2 release there were [167 commits][commits] by -[cjmctague], [ctubbsii], [Garvit244], [keith-turner], [mikewalch]. Information about all --- End diff -- can remove these people links at the end of the doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-fluo-website pull request #23: Fluo 1.0.0
Github user keith-turner commented on a diff in the pull request: https://github.com/apache/incubator-fluo-website/pull/23#discussion_r83219527 --- Diff: _posts/blog/2016-09-27-1.0.0-incubating-is-released.md --- @@ -0,0 +1,12 @@ +--- +title: "Fluo 1.0.0-incubating is released!" +date: 2016-09-27 15:30:00 +--- + +Apache Fluo is happy to announce a [1.0.0-incubating][rel] release. This is the first +release of Fluo as an Apache project and the fourth release for the project. Starting +with this release, Fluo will follow [semver][semver] for all future API changes. See +the [release page][rel] for more information. --- End diff -- This last sentence seems redundant. since link is also in 1st sentence. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
