http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2016-10-04-1.0.0-incubating.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2016-10-04-1.0.0-incubating.md b/_posts/release-notes/2016-10-04-1.0.0-incubating.md deleted file mode 100644 index 87a01aa..0000000 --- a/_posts/release-notes/2016-10-04-1.0.0-incubating.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: Fluo 1.0.0-incubating Release Notes -date: 2016-10-04 15:30:00 -version: 1.0.0-incubating ---- - -Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth -release for the project. Since the beta-2 release there were [167 commits][commits] by -[cjmctague], [ctubbsii], [Garvit244], [keith-turner], [mikewalch]. Information about all -contributors since the beggining is available on the [people page](/people/). For information about -changes before this release see the [archive](/release-notes/). - -## API Changes since Beta 2 - -Starting with this release, Apache Fluo's API will follow [semver]. The API is defined as -everything under the org.apache.fluo.api Java package. Given that the plan is to support this API -for a long time, a lot of API improvements were made since the beta-2 release. The following is a -list of significant API changes since beta-2. - - * [#772][772] : Get with default - * [#770][770] : Use varargs for Columns - * [#768][768] : Made Obsever and Application configuration consistent - * [#743][743] : Make data classes final - * [#723][723] : Removed FluoAdmin.InitOps() - * [#714][714] : Use Charsequence in API - * [#695][695] : Removed all 3rd party types from API. - * [#680][680] : Changed package prefix from io.fluo to org.apache - * [#647][647] : Added Oracle and Worker to API - * [#639][639] : Improved scanning API - * [#626][626] : Added operation to get row+column pairs. Further improved by [#758][758]. - * [#119][119] - Consistently offer String in API - -## Significant Changes since Beta 2 - -The previous releases of Fluo contained specific versions of Accumulo, Hadoop, Zookeeper, and other -Fluo dependencies. However, the actual dependencies needed should be determined by the existing -version of software where Fluo is run. Fluo changed to a model of making its classpath be -completely configurable by an administrator. Example classpath configuration and download scripts -ship with Fluo to help make setup easier. These script were only tested with specific versions of -Hadoop and Accumulo and may not work with all versions. These changes were made in [#706][706] and -[#687][687]. - -Added a version command to the Fluo scripts in [#779][779]. This makes it easy to automatically -build shaded jars with the correct version of Fluo for use in Spark and Map Reduce. - -Two major improvements related to metrics were made. Support for user metrics was added by -[#767][767]. This allows observers to easily report application specific metrics. In [#635][635] -the example InfluxDB and Grafana configurations were updated to show historical metrics. - -Transaction processing throughput was increased by changes in [#593][593] and some subsequent bug -fixes. These changes moved commit processing from the threads executing user code to an -asynchronous background process. This allows user threads to queue a transaction for commit an -immediately start working on another notification. This new process allows many more transactions -to be working through the commit process concurrently. Before the change a worker could only be -committing up to a few hundred transactions at any time. After the change, a single worker could -easily be committing tens of thousands of transactions at a time. These changes increase the latency of -individual transactions, which can have some drawbacks [#650][650]. For this release changes were -made in [#654][654] to use a priority queue for notifications in the worker. The changes in -[#654][654] partially solves the lock wait problem identified in [#650][650]. - -Fluo utilizes server side Accumulo iterators. For this release these iterators were modified to -leverage seeking on the sever side. Before these changes the iterators used to sequentially scan -data. For popular cells that had a lot versions, this could be slow. These changes were made in -[#623][623]. - -## Other Issues of Interest - -* [#668][668] - Added method to append byte to BytesBuilder -* [#648][648] - Added documentation to help users if YARN is killing containers -* [#619][619] [#621][621] - Created Fluo logo - -## Testing - -A two day run of Webindex was started on September 7th using the latest snapshot version of Fluo. -This test used 11 EC2 m3.xlarge nodes. The test ran without problem. A good bit before that a [3 -day run](/blog/2016/05/17/webindex-long-run-2/) of Webindex was conducted. - -[semver]: http://semver.org/ -[commits]: https://github.com/apache/incubator-fluo/compare/1.0.0-beta-2...e1dbc60 -[ctubbsii]: https://github.com/ctubbsii -[mikewalch]: https://github.com/mikewalch -[keith-turner]: https://github.com/keith-turner -[cjmctague]: https://github.com/cjmctague -[Garvit244]: https://github.com/Garvit244 - -[779]: https://github.com/fluo-io/fluo/issues/779 -[772]: https://github.com/fluo-io/fluo/issues/772 -[770]: https://github.com/fluo-io/fluo/issues/770 -[768]: https://github.com/fluo-io/fluo/issues/768 -[767]: https://github.com/fluo-io/fluo/issues/767 -[758]: https://github.com/fluo-io/fluo/issues/758 -[743]: https://github.com/fluo-io/fluo/issues/743 -[723]: https://github.com/fluo-io/fluo/issues/723 -[714]: https://github.com/fluo-io/fluo/issues/714 -[706]: https://github.com/fluo-io/fluo/issues/706 -[695]: https://github.com/fluo-io/fluo/issues/695 -[687]: https://github.com/fluo-io/fluo/issues/687 -[680]: https://github.com/fluo-io/fluo/issues/680 -[668]: https://github.com/fluo-io/fluo/issues/668 -[654]: https://github.com/fluo-io/fluo/issues/654 -[650]: https://github.com/fluo-io/fluo/issues/650 -[648]: https://github.com/fluo-io/fluo/issues/648 -[647]: https://github.com/fluo-io/fluo/issues/647 -[639]: https://github.com/fluo-io/fluo/issues/639 -[635]: https://github.com/fluo-io/fluo/issues/635 -[626]: https://github.com/fluo-io/fluo/issues/626 -[623]: https://github.com/fluo-io/fluo/issues/623 -[621]: https://github.com/fluo-io/fluo/issues/621 -[619]: https://github.com/fluo-io/fluo/issues/619 -[593]: https://github.com/fluo-io/fluo/issues/593 -[119]: https://github.com/fluo-io/fluo/issues/119 -
http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-summaries/2014-10-02-1.0.0-alpha-1.md ---------------------------------------------------------------------- diff --git a/_posts/release-summaries/2014-10-02-1.0.0-alpha-1.md b/_posts/release-summaries/2014-10-02-1.0.0-alpha-1.md deleted file mode 100644 index c660b0a..0000000 --- a/_posts/release-summaries/2014-10-02-1.0.0-alpha-1.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: Fluo 1.0.0-alpha-1 Release Summary -date: 2014-10-02 15:30:00 -version: 1.0.0-alpha-1 -historical: true -redirect_from: /1.0.0-alpha-1-release/ ---- - -The [1.0.0-alpha-1][release] release is the first release of Fluo. It is not ready for production use. See the [release notes][notes] for -significant features. [Javadocs][apidocs] are also available. - -## Quickstart - -If you are new to Fluo, the best way to get started is to follow the [quickstart] which starts a local Fluo instance -(called MiniFluo). By using MiniFluo, you can avoid following the steps below to run your own Fluo instance. - -## Running a Fluo instance - -In order to run a Fluo instance, you will need [Accumulo] (version 1.6+), [Hadoop] (version 2.2+), and [Zookeeper] installed and -running on your local machine or cluster. - -Fluo distributions are built for specific releases of Hadoop and Accumulo. If you are using Accumulo 1.6.1 and Hadoop 2.3.0, you -can download the [1.0.0-alpha-1 tarball][release]. Otherwise, you will need to clone the Fluo repo and build a distribution tarball -of 1.0.0-alpha-1 for your environment: - -{% highlight bash %} -git clone https://github.com/fluo-io/fluo.git -cd fluo/ -git checkout 1.0.0-alpha-1 -mvn package -Daccumulo.version=1.6.1-SNAPSHOT -Dhadoop.version=2.4.0 -{% endhighlight %} - -The distribution tarball will be created in ```modules/distribution/target``` - -When you have a Fluo distribution, install it to a directory: - -{% highlight bash %} -tar xzf fluo-distribution-1.0.0-alpha-1-bin.tar.gz -{% endhighlight %} - -Next, follow the instructions to configure and run Fluo in the [README]. - -## Running Fluo applications - -Once you have Fluo installed and running on your cluster, you can now run Fluo applications. Fluo applications consist of -clients and observers. If you are new to Fluo, consider first building and running the [phrasecount] application on your Fluo -instance. Otherwise, you will need to include the following in your Maven pom and follow the steps below to create your own -Fluo client or observer: - -{% highlight xml %} -<dependency> - <groupId>io.fluo</groupId> - <artifactId>fluo-api</artifactId> - <version>1.0.0-alpha-1</version> -</dependency> -<dependency> - <groupId>io.fluo</groupId> - <artifactId>fluo-core</artifactId> - <version>1.0.0-alpha-1</version> - <scope>runtime</scope> -</dependency> -{% endhighlight %} - -### Creating a Fluo client - -To create a [FluoClient], you will need all properties with the ```io.fluo.client.*``` prefix set in the configuration that is passed -to the ```newClient()``` method of [FluoFactory]. See [fluo.properties] for details on these properties. - -### Creating a Fluo observer - -To create an observer, follow these steps: - -1. Create a class that extends [AbstractObserver]. -2. Build a jar containing this class and include this jar in ```lib/observers``` of your Fluo installation. -3. Configure your Fluo instance to use this observer by modifying the Observer section of [fluo.properties]. -4. Restart your Fluo instance so that your Fluo workers load the new observer. - -[apidocs]: /apidocs/fluo/1.0.0-alpha-1/ -[quickstart]: https://github.com/fluo-io/fluo-quickstart -[notes]: /release-notes/1.0.0-alpha-1/ -[Accumulo]: https://accumulo.apache.org/ -[Hadoop]: http://hadoop.apache.org/ -[Zookeeper]: http://zookeeper.apache.org/ -[release]: https://github.com/fluo-io/fluo/releases/tag/1.0.0-alpha-1 -[README]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/README.md -[phrasecount]: https://github.com/fluo-io/phrasecount -[FluoFactory]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/api/src/main/java/io/fluo/api/client/FluoFactory.java -[FluoClient]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/api/src/main/java/io/fluo/api/client/FluoClient.java -[fluo.properties]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/distribution/src/main/config/fluo.properties -[AbstractObserver]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/api/src/main/java/io/fluo/api/observer/AbstractObserver.java http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-summaries/2015-06-09-1.0.0-beta-1.md ---------------------------------------------------------------------- diff --git a/_posts/release-summaries/2015-06-09-1.0.0-beta-1.md b/_posts/release-summaries/2015-06-09-1.0.0-beta-1.md deleted file mode 100644 index a56886b..0000000 --- a/_posts/release-summaries/2015-06-09-1.0.0-beta-1.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: Fluo 1.0.0-beta-1 Release Summary -date: 2015-06-09 18:30:00 -version: 1.0.0-beta-1 -historical: true -redirect_from: /1.0.0-beta-1-release/ ---- - -The [1.0.0-beta-1][release] release is the second release of Fluo. It contains the following significant improvements: - - * Simplified administration of Fluo on a local machine or EC2 cluster - * Transaction metrics are now viewable using common monitoring tools - * Improved processing of notifications in Fluo workers - * Easier management of Fluo from the command line - * Support for running multiple Fluo applications on a single cluster - * Improved the deletion of observer notifications - -This release is not recommended for production use. There is no upgrade path from 1.0.0-alpha-1 to 1.0.0-beta-1. - -Below are resources for this release: - - * Read the [release notes][notes] for a full list of notable features and bug fixes. - * View the [documentation] for help getting started with Fluo. - * Download the [Fluo binary tarball][release] for 1.0.0-beta-1 from GitHub. - * [Javadocs][apidocs] are available for this release. - * A [tag] of Fluo codebase for 1.0.0-beta-1 is available. - * The [Quickstart] and [Phrasecount] applications were updated to work with this release. - -[release]: https://github.com/fluo-io/fluo/releases/tag/1.0.0-beta-1 -[notes]: /release-notes/1.0.0-beta-1/ -[apidocs]: /apidocs/fluo/1.0.0-beta-1/ -[documentation]: /docs/fluo/1.0.0-beta-1/ -[tag]: https://github.com/fluo-io/fluo/tree/1.0.0-beta-1 -[Quickstart]: https://github.com/fluo-io/fluo-quickstart -[Phrasecount]: https://github.com/fluo-io/phrasecount http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-summaries/2016-01-12-1.0.0-beta-2.md ---------------------------------------------------------------------- diff --git a/_posts/release-summaries/2016-01-12-1.0.0-beta-2.md b/_posts/release-summaries/2016-01-12-1.0.0-beta-2.md deleted file mode 100644 index 7125c99..0000000 --- a/_posts/release-summaries/2016-01-12-1.0.0-beta-2.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -title: Fluo 1.0.0-beta-2 Release Summary -date: 2016-01-12 15:30:00 -version: 1.0.0-beta-2 -historical: true -redirect_from: /1.0.0-beta-2-release/ ---- - -The [1.0.0-beta-2][release] release is the third release of Fluo and likely the final -release before 1.0.0. - -It contains the following significant improvements: - - * Fluo metrics were improved and can now be sent to InfluxDB and viewed in Grafana. - * Improvements were made to Fluo to make it work better with Apache Spark. - * Several performances improvements were made to decrease transaction commit time. - * Fluo administration was improved with bug fixes and the addition of new commands. - * Several improvements were made to the Fluo API. - -Fluo is now at a point where its two cluster test suites, [Webindex][webindex] -and [Stress][fluo-stress], are running well for long periods on EC2. - -Below are resources for this release: - - * Read the [release notes][notes] for a full list of notable features and bug fixes. - * View the [documentation][docs] for help getting started with Fluo. - * Download the [Fluo binary tarball][release] for 1.0.0-beta-2 from GitHub. - * [Javadocs][apidocs] are available for this release. - * A [tag] of Fluo codebase for 1.0.0-beta-2 is available. - * The [Quickstart] and [Phrasecount] applications were updated to work with this release. - -[release]: https://github.com/fluo-io/fluo/releases/tag/1.0.0-beta-2 -[notes]: /release-notes/1.0.0-beta-2/ -[apidocs]: /apidocs/fluo/1.0.0-beta-2/ -[docs]: /docs/fluo/1.0.0-beta-2/ -[tag]: https://github.com/fluo-io/fluo/tree/1.0.0-beta-2 -[Quickstart]: https://github.com/fluo-io/fluo-quickstart -[Phrasecount]: https://github.com/fluo-io/phrasecount -[webindex]: https://github.com/fluo-io/webindex -[fluo-stress]: https://github.com/fluo-io/fluo-stress http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release/2014-10-02-fluo-1.0.0-alpha-1.md ---------------------------------------------------------------------- diff --git a/_posts/release/2014-10-02-fluo-1.0.0-alpha-1.md b/_posts/release/2014-10-02-fluo-1.0.0-alpha-1.md new file mode 100644 index 0000000..cba0e24 --- /dev/null +++ b/_posts/release/2014-10-02-fluo-1.0.0-alpha-1.md @@ -0,0 +1,90 @@ +--- +title: Fluo 1.0.0-alpha-1 Release +date: 2014-10-02 15:30:00 +version: fluo-1.0.0-alpha-1 +historical: true +redirect_from: /1.0.0-alpha-1-release/ +--- + +The [1.0.0-alpha-1][release] release is the first release of Fluo. It is not ready for production use. See the [release notes][notes] for +significant features. [Javadocs][apidocs] are also available. + +## Quickstart + +If you are new to Fluo, the best way to get started is to follow the [quickstart] which starts a local Fluo instance +(called MiniFluo). By using MiniFluo, you can avoid following the steps below to run your own Fluo instance. + +## Running a Fluo instance + +In order to run a Fluo instance, you will need [Accumulo] (version 1.6+), [Hadoop] (version 2.2+), and [Zookeeper] installed and +running on your local machine or cluster. + +Fluo distributions are built for specific releases of Hadoop and Accumulo. If you are using Accumulo 1.6.1 and Hadoop 2.3.0, you +can download the [1.0.0-alpha-1 tarball][release]. Otherwise, you will need to clone the Fluo repo and build a distribution tarball +of 1.0.0-alpha-1 for your environment: + +{% highlight bash %} +git clone https://github.com/fluo-io/fluo.git +cd fluo/ +git checkout 1.0.0-alpha-1 +mvn package -Daccumulo.version=1.6.1-SNAPSHOT -Dhadoop.version=2.4.0 +{% endhighlight %} + +The distribution tarball will be created in ```modules/distribution/target``` + +When you have a Fluo distribution, install it to a directory: + +{% highlight bash %} +tar xzf fluo-distribution-1.0.0-alpha-1-bin.tar.gz +{% endhighlight %} + +Next, follow the instructions to configure and run Fluo in the [README]. + +## Running Fluo applications + +Once you have Fluo installed and running on your cluster, you can now run Fluo applications. Fluo applications consist of +clients and observers. If you are new to Fluo, consider first building and running the [phrasecount] application on your Fluo +instance. Otherwise, you will need to include the following in your Maven pom and follow the steps below to create your own +Fluo client or observer: + +{% highlight xml %} +<dependency> + <groupId>io.fluo</groupId> + <artifactId>fluo-api</artifactId> + <version>1.0.0-alpha-1</version> +</dependency> +<dependency> + <groupId>io.fluo</groupId> + <artifactId>fluo-core</artifactId> + <version>1.0.0-alpha-1</version> + <scope>runtime</scope> +</dependency> +{% endhighlight %} + +### Creating a Fluo client + +To create a [FluoClient], you will need all properties with the ```io.fluo.client.*``` prefix set in the configuration that is passed +to the ```newClient()``` method of [FluoFactory]. See [fluo.properties] for details on these properties. + +### Creating a Fluo observer + +To create an observer, follow these steps: + +1. Create a class that extends [AbstractObserver]. +2. Build a jar containing this class and include this jar in ```lib/observers``` of your Fluo installation. +3. Configure your Fluo instance to use this observer by modifying the Observer section of [fluo.properties]. +4. Restart your Fluo instance so that your Fluo workers load the new observer. + +[apidocs]: /apidocs/fluo/1.0.0-alpha-1/ +[quickstart]: https://github.com/fluo-io/fluo-quickstart +[notes]: /release-notes/fluo-1.0.0-alpha-1/ +[Accumulo]: https://accumulo.apache.org/ +[Hadoop]: http://hadoop.apache.org/ +[Zookeeper]: http://zookeeper.apache.org/ +[release]: https://github.com/fluo-io/fluo/releases/tag/1.0.0-alpha-1 +[README]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/README.md +[phrasecount]: https://github.com/fluo-io/phrasecount +[FluoFactory]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/api/src/main/java/io/fluo/api/client/FluoFactory.java +[FluoClient]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/api/src/main/java/io/fluo/api/client/FluoClient.java +[fluo.properties]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/distribution/src/main/config/fluo.properties +[AbstractObserver]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/api/src/main/java/io/fluo/api/observer/AbstractObserver.java http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release/2015-06-09-fluo-1.0.0-beta-1.md ---------------------------------------------------------------------- diff --git a/_posts/release/2015-06-09-fluo-1.0.0-beta-1.md b/_posts/release/2015-06-09-fluo-1.0.0-beta-1.md new file mode 100644 index 0000000..99fa56b --- /dev/null +++ b/_posts/release/2015-06-09-fluo-1.0.0-beta-1.md @@ -0,0 +1,35 @@ +--- +title: Fluo 1.0.0-beta-1 Release +date: 2015-06-09 18:30:00 +version: fluo-1.0.0-beta-1 +historical: true +redirect_from: /1.0.0-beta-1-release/ +--- + +The [1.0.0-beta-1][release] release is the second release of Fluo. It contains the following significant improvements: + + * Simplified administration of Fluo on a local machine or EC2 cluster + * Transaction metrics are now viewable using common monitoring tools + * Improved processing of notifications in Fluo workers + * Easier management of Fluo from the command line + * Support for running multiple Fluo applications on a single cluster + * Improved the deletion of observer notifications + +This release is not recommended for production use. There is no upgrade path from 1.0.0-alpha-1 to 1.0.0-beta-1. + +Below are resources for this release: + + * Read the [release notes][notes] for a full list of notable features and bug fixes. + * View the [documentation] for help getting started with Fluo. + * Download the [Fluo binary tarball][release] for 1.0.0-beta-1 from GitHub. + * [Javadocs][apidocs] are available for this release. + * A [tag] of Fluo codebase for 1.0.0-beta-1 is available. + * The [Quickstart] and [Phrasecount] applications were updated to work with this release. + +[release]: https://github.com/fluo-io/fluo/releases/tag/1.0.0-beta-1 +[notes]: /release-notes/fluo-1.0.0-beta-1/ +[apidocs]: /apidocs/fluo/1.0.0-beta-1/ +[documentation]: /docs/fluo/1.0.0-beta-1/ +[tag]: https://github.com/fluo-io/fluo/tree/1.0.0-beta-1 +[Quickstart]: https://github.com/fluo-io/fluo-quickstart +[Phrasecount]: https://github.com/fluo-io/phrasecount http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release/2016-01-12-fluo-1.0.0-beta-2.md ---------------------------------------------------------------------- diff --git a/_posts/release/2016-01-12-fluo-1.0.0-beta-2.md b/_posts/release/2016-01-12-fluo-1.0.0-beta-2.md new file mode 100644 index 0000000..c6cc6d6 --- /dev/null +++ b/_posts/release/2016-01-12-fluo-1.0.0-beta-2.md @@ -0,0 +1,40 @@ +--- +title: Fluo 1.0.0-beta-2 Release +date: 2016-01-12 15:30:00 +version: fluo-1.0.0-beta-2 +historical: true +redirect_from: /1.0.0-beta-2-release/ +--- + +The [1.0.0-beta-2][release] release is the third release of Fluo and likely the final +release before 1.0.0. + +It contains the following significant improvements: + + * Fluo metrics were improved and can now be sent to InfluxDB and viewed in Grafana. + * Improvements were made to Fluo to make it work better with Apache Spark. + * Several performances improvements were made to decrease transaction commit time. + * Fluo administration was improved with bug fixes and the addition of new commands. + * Several improvements were made to the Fluo API. + +Fluo is now at a point where its two cluster test suites, [Webindex][webindex] +and [Stress][fluo-stress], are running well for long periods on EC2. + +Below are resources for this release: + + * Read the [release notes][notes] for a full list of notable features and bug fixes. + * View the [documentation][docs] for help getting started with Fluo. + * Download the [Fluo binary tarball][release] for 1.0.0-beta-2 from GitHub. + * [Javadocs][apidocs] are available for this release. + * A [tag] of Fluo codebase for 1.0.0-beta-2 is available. + * The [Quickstart] and [Phrasecount] applications were updated to work with this release. + +[release]: https://github.com/fluo-io/fluo/releases/tag/1.0.0-beta-2 +[notes]: /release-notes/fluo-1.0.0-beta-2/ +[apidocs]: /apidocs/fluo/1.0.0-beta-2/ +[docs]: /docs/fluo/1.0.0-beta-2/ +[tag]: https://github.com/fluo-io/fluo/tree/1.0.0-beta-2 +[Quickstart]: https://github.com/fluo-io/fluo-quickstart +[Phrasecount]: https://github.com/fluo-io/phrasecount +[webindex]: https://github.com/fluo-io/webindex +[fluo-stress]: https://github.com/fluo-io/fluo-stress http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release/2016-01-13-fluo-recipes-1.0.0-beta-1.md ---------------------------------------------------------------------- diff --git a/_posts/release/2016-01-13-fluo-recipes-1.0.0-beta-1.md b/_posts/release/2016-01-13-fluo-recipes-1.0.0-beta-1.md new file mode 100644 index 0000000..6d82aff --- /dev/null +++ b/_posts/release/2016-01-13-fluo-recipes-1.0.0-beta-1.md @@ -0,0 +1,19 @@ +--- +title: Fluo Recipes 1.0.0-beta-1 release +date: 2016-01-13 15:30:00 +version: fluo-recipes-1.0.0-beta-1 +historical: true +--- + +This is the first release of Fluo Recipes which provides common code for Fluo developers. + +The jars for this release of Fluo Recipes are available in [Maven Central][central]. + +Below are resources for this release: + +* [Documentation][docs] +* [Javadocs][api] + +[central]: http://search.maven.org/#search|ga|1|fluo-recipes +[docs]: /docs/fluo-recipes/1.0.0-beta-1 +[api]: /apidocs/fluo-recipes/1.0.0-beta-1 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release/2016-03-29-fluo-recipes-1.0.0-beta-2.md ---------------------------------------------------------------------- diff --git a/_posts/release/2016-03-29-fluo-recipes-1.0.0-beta-2.md b/_posts/release/2016-03-29-fluo-recipes-1.0.0-beta-2.md new file mode 100644 index 0000000..fd96f1e --- /dev/null +++ b/_posts/release/2016-03-29-fluo-recipes-1.0.0-beta-2.md @@ -0,0 +1,19 @@ +--- +title: Fluo Recipes 1.0.0-beta-2 release +date: 2016-03-29 15:30:00 +version: fluo-recipes-1.0.0-beta-2 +historical: true +--- + +This is the second release of Fluo Recipes which provides common code for Fluo developers. + +The jars for this release of Fluo Recipes are available in [Maven Central][central]. + +Below are resources for this release: + +* [Documentation][docs] +* [Javadocs][api] + +[central]: http://search.maven.org/#search|ga|1|fluo-recipes +[docs]: /docs/fluo-recipes/1.0.0-beta-2 +[api]: /apidocs/fluo-recipes/1.0.0-beta-2 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release/2016-09-27-fluo-1.0.0-incubating.md ---------------------------------------------------------------------- diff --git a/_posts/release/2016-09-27-fluo-1.0.0-incubating.md b/_posts/release/2016-09-27-fluo-1.0.0-incubating.md new file mode 100644 index 0000000..2d581b7 --- /dev/null +++ b/_posts/release/2016-09-27-fluo-1.0.0-incubating.md @@ -0,0 +1,27 @@ +--- +title: Fluo 1.0.0-incubating release +date: 2016-09-27 15:30:00 +version: fluo-1.0.0-incubating +--- + +Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth +release for the project. Below are resources for this release: + + * Download release: [fluo-1.0.0-incubating-bin.tgz][binary] + * Verify release: [ASC][asc] [MD5][md5] [SHA1][sha1] + * [Release notes][notes] + * [Documentation][docs] + * [Javadocs][apidocs] + +Starting with 1.0.0-incubating, Apache Fluo will follow [semver](http://semver.org/) for all API +changes. The API consists of everything under the **org.apache.fluo.api** package. Code outside of this +package can change at any time. If your project is using Fluo code that falls outside of the API, +then consider [initiating a discussion](/getinvolved/) about adding it to the API. + +[binary]: https://www.apache.org/dyn/closer.lua/incubator/fluo/fluo/1.0.0-incubating/fluo-1.0.0-incubating-bin.tar.gz +[asc]: https://www.apache.org/dist/incubator/fluo/fluo/1.0.0-incubating/fluo-1.0.0-incubating-bin.tar.gz.asc +[md5]: https://www.apache.org/dist/incubator/fluo/fluo/1.0.0-incubating/MD5SUM +[sha1]: https://www.apache.org/dist/incubator/fluo/fluo/1.0.0-incubating/SHA1SUM +[notes]: /release-notes/fluo-1.0.0-incubating/ +[apidocs]: {{ site.fluo_api_base }}/1.0.0-incubating/ +[docs]: /docs/fluo/1.0.0-incubating/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_scripts/convert-docs.py ---------------------------------------------------------------------- diff --git a/_scripts/convert-docs.py b/_scripts/convert-docs.py index dc9a766..dc38d40 100755 --- a/_scripts/convert-docs.py +++ b/_scripts/convert-docs.py @@ -14,6 +14,8 @@ output_dir = sys.argv[2] args = output_dir.rpartition("/docs") url_prefix = args[1] + args[2] +print url_prefix +print args release_ver = url_prefix.split("/")[3] github_prefix = "https://github.com/apache/fluo/blob/{0}/modules/".format(release_ver) apidocs_prefix = "/apidocs/fluo/{0}/".format(release_ver) @@ -32,6 +34,10 @@ def convert_file(inPath, outPath): print "Creating ", outPath with open(inPath) as fin: + # skip license + for x in range(0, 16): + fin.readline() + title = fin.readline().strip() fin.readline() http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/apidocs/archive.md ---------------------------------------------------------------------- diff --git a/apidocs/archive.md b/apidocs/archive.md new file mode 100644 index 0000000..2ec0f11 --- /dev/null +++ b/apidocs/archive.md @@ -0,0 +1,23 @@ +--- +layout: page +title: Pre-ASF API Archive +--- + +<div class="alert alert-danger" role="alert">Please note - This page links to Javadocs for non-ASF releases of Fluo. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> + +#### Fluo API documentation (before Apache) + +* [1.0.0-beta-2][fluo-b2] - January 12, 2016 +* [1.0.0-beta-1][fluo-b1] - June 9, 2015 +* [1.0.0-alpha-1][fluo-a1] - October 2, 2014 + +#### Fluo Recipes API documentation (before Apache) + +* [1.0.0-beta-2][recipes-b2] - March 19, 2016 +* [1.0.0-beta-1][recipes-b1] - January 13, 2016 + +[fluo-b2]: /apidocs/fluo/1.0.0-beta-2/ +[fluo-b1]: /apidocs/fluo/1.0.0-beta-1/ +[fluo-a1]: /apidocs/fluo/1.0.0-alpha-1/ +[recipes-b2]: /apidocs/fluo-recipes/1.0.0-beta-2/ +[recipes-b1]: /apidocs/fluo-recipes/1.0.0-beta-1/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/apidocs/fluo-recipes/index.md ---------------------------------------------------------------------- diff --git a/apidocs/fluo-recipes/index.md b/apidocs/fluo-recipes/index.md deleted file mode 100644 index db8bcd0..0000000 --- a/apidocs/fluo-recipes/index.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -layout: page -title: Fluo Recipes API Archive ---- - -<div class="alert alert-danger" role="alert">Please note - This page links to Javadocs for non-ASF releases of Fluo. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> - -Fluo Recipes API documentation before Apache incubation: - -* [1.0.0-beta-2][recipes-b2] - March 19, 2016 -* [1.0.0-beta-1][recipes-b1] - January 13, 2016 - -[recipes-b2]: /apidocs/fluo-recipes/1.0.0-beta-2/ -[recipes-b1]: /apidocs/fluo-recipes/1.0.0-beta-1/ - http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/apidocs/fluo/archive.md ---------------------------------------------------------------------- diff --git a/apidocs/fluo/archive.md b/apidocs/fluo/archive.md deleted file mode 100644 index b7af010..0000000 --- a/apidocs/fluo/archive.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -layout: page -title: Fluo API Archive ---- - -<div class="alert alert-danger" role="alert">Please note - This page links to Javadocs for non-ASF releases of Fluo. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> - -Fluo API documentation before Apache incubation: - -* [1.0.0-beta-2][fluo-b2] - January 12, 2016 -* [1.0.0-beta-1][fluo-b1] - June 9, 2015 -* [1.0.0-alpha-1][fluo-a1] - October 2, 2014 - -[fluo-b2]: /apidocs/fluo/1.0.0-beta-2/ -[fluo-b1]: /apidocs/fluo/1.0.0-beta-1/ -[fluo-a1]: /apidocs/fluo/1.0.0-alpha-1/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/apidocs/fluo/index.md ---------------------------------------------------------------------- diff --git a/apidocs/fluo/index.md b/apidocs/fluo/index.md deleted file mode 100644 index 67c7777..0000000 --- a/apidocs/fluo/index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -layout: page -title: Apache Fluo API ---- - - -Apache Fluo API documentation : - - * [1.0.0-incubating][fluo-1.0] - October 4, 2016 - -Documentation relating to release made before joining the ASF are [archived](archive). - -[fluo-1.0]: /apidocs/fluo/1.0.0-incubating/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/apidocs/index.md ---------------------------------------------------------------------- diff --git a/apidocs/index.md b/apidocs/index.md index 74122f5..93de575 100644 --- a/apidocs/index.md +++ b/apidocs/index.md @@ -1,19 +1,22 @@ --- layout: page title: API Documentation +redirect_from: + - /apidocs/fluo/ + - /apidocs/fluo-recipes/ --- -<div class="alert alert-danger" role="alert">Please note - This page links to Javadocs for non-ASF releases of Fluo. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> +[Apache Fluo] and [Apache Fluo Recipes] have seperate API documentation as they are different repositories with their own release cycle. +#### Apache Fluo API -[Apache Fluo] and [Apache Fluo Recipes] have seperate API documentation as they are different repositories with their own release cycle. +* <a href="{{ site.fluo_api_base }}/1.0.0-incubating/" target="_blank">1.0.0-incubating</a> - September 27, 2016 + +#### Apache Fluo Recipes API + +* The first release of Apache Fluo Recipes has not been made yet. -* Fluo API documentation - [Latest][fluo-api-latest] \| [Archive][fluo-api-archive] -* Fluo Recipes API docoumentation - [Latest][recipes-api-latest] \| [Archive][recipes-api-archive] +API Documentation for releases before joining Apache are available in the [Pre-ASF archive](archive). [Apache Fluo]: https://github.com/apache/fluo [Apache Fluo Recipes]: https://github.com/apache/fluo-recipes -[fluo-api-latest]: /apidocs/fluo/{{ site.latest_fluo_release }}/ -[recipes-api-latest]: /apidocs/fluo-recipes/{{ site.latest_recipes_release }}/ -[fluo-api-archive]: /apidocs/fluo/ -[recipes-api-archive]: /apidocs/fluo-recipes/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/archive.md ---------------------------------------------------------------------- diff --git a/docs/archive.md b/docs/archive.md new file mode 100644 index 0000000..e55389c --- /dev/null +++ b/docs/archive.md @@ -0,0 +1,23 @@ +--- +layout: page +title: Pre-ASF Documentation Archive +--- + +<div class="alert alert-danger" role="alert">Please note - This page links to documentation for non-ASF releases of Fluo. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> + +#### Fluo documentation (before Apache) + +* [1.0.0-beta-2][fluo-b2] - January 12, 2016 +* [1.0.0-beta-1][fluo-b1] - June 9, 2015 +* [1.0.0-alpha-1][fluo-a1] - October 2, 2014 + +#### Fluo Recipes documentation (before Apache) + +* [1.0.0-beta-2][recipes-b2] - March 19, 2016 +* [1.0.0-beta-1][recipes-b1] - January 13, 2016 + +[fluo-b2]: /docs/fluo/1.0.0-beta-2/ +[fluo-b1]: /docs/fluo/1.0.0-beta-1/ +[fluo-a1]: /docs/fluo/1.0.0-alpha-1/ +[recipes-b2]: /docs/fluo-recipes/1.0.0-beta-2/ +[recipes-b1]: /docs/fluo-recipes/1.0.0-beta-1/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo-recipes/index.md ---------------------------------------------------------------------- diff --git a/docs/fluo-recipes/index.md b/docs/fluo-recipes/index.md deleted file mode 100644 index e32204c..0000000 --- a/docs/fluo-recipes/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -layout: page -title: Fluo Recipes Documentation Archive ---- - -Fluo Recipes documentation before Apache incubation: - -* [1.0.0-beta-2][recipes-b2] - March 19, 2016 -* [1.0.0-beta-1][recipes-b1] - January 13, 2016 - -[recipes-b2]: /docs/fluo-recipes/1.0.0-beta-2/ -[recipes-b1]: /docs/fluo-recipes/1.0.0-beta-1/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-beta-1/contributing.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-beta-1/contributing.md b/docs/fluo/1.0.0-beta-1/contributing.md index 7409a1b..4ae3f71 100644 --- a/docs/fluo/1.0.0-beta-1/contributing.md +++ b/docs/fluo/1.0.0-beta-1/contributing.md @@ -7,7 +7,7 @@ version: 1.0.0-beta-1 Building Fluo ------------- -If you have [Git], [Maven], and [Java](version 7+) installed, run these commands +If you have [Git], [Maven], and [Java][java] (version 7+) installed, run these commands to build Fluo: git clone https://github.com/fluo-io/fluo.git @@ -25,6 +25,6 @@ a local Fluo instance (called MiniFluo) and run against it. * A [Stress test][Stress] application designed to run on a cluster. [Git]: http://git-scm.com/ -[Java]: https://www.oracle.com/java/index.html +[java]: https://www.oracle.com/java/index.html [Maven]: http://maven.apache.org/ [Stress]: https://github.com/fluo-io/fluo-stress http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-beta-2/contributing.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-beta-2/contributing.md b/docs/fluo/1.0.0-beta-2/contributing.md index dc15baf..6ff613a 100644 --- a/docs/fluo/1.0.0-beta-2/contributing.md +++ b/docs/fluo/1.0.0-beta-2/contributing.md @@ -7,7 +7,7 @@ version: 1.0.0-beta-2 Building Fluo ------------- -If you have [Git], [Maven], and [Java](version 7+) installed, run these commands +If you have [Git], [Maven], and [Java][java] (version 7+) installed, run these commands to build Fluo: git clone https://github.com/fluo-io/fluo.git @@ -25,6 +25,6 @@ a local Fluo instance (called MiniFluo) and run against it. * A [Stress test][Stress] application designed to run on a cluster. [Git]: http://git-scm.com/ -[Java]: https://www.oracle.com/java/index.html +[java]: https://www.oracle.com/java/index.html [Maven]: http://maven.apache.org/ [Stress]: https://github.com/fluo-io/fluo-stress http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/applications.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/applications.md b/docs/fluo/1.0.0-incubating/applications.md new file mode 100644 index 0000000..388af66 --- /dev/null +++ b/docs/fluo/1.0.0-incubating/applications.md @@ -0,0 +1,167 @@ +--- +layout: fluo-doc +title: Fluo Applications +version: 1.0.0-incubating +--- + +Once you have Fluo installed and running on your cluster, you can now run Fluo applications which +consist of clients and observers. + +For both clients and observers, you will need to include the following in your Maven pom: + +```xml +<dependency> + <groupId>org.apache.fluo</groupId> + <artifactId>fluo-api</artifactId> + <version>1.0.0-incubating</version> +</dependency> +<dependency> + <groupId>org.apache.fluo</groupId> + <artifactId>fluo-core</artifactId> + <version>1.0.0-incubating</version> + <scope>runtime</scope> +</dependency> +``` + +Fluo provides a classpath command to help users build a runtime classpath. This command along with +the `hadoop jar` command is useful when writing scripts to run Fluo client code. These commands +allow the scripts to use the versions of Hadoop, Accumulo, and Zookeeper installed on a cluster. + +## Creating a Fluo client + +To create a [FluoClient], you will need to provide it with a [FluoConfiguration] object that is +configured to connect to your Fluo instance. + +If you have access to the [fluo.properties] file that was used to configure your Fluo instance, you +can use it to build a [FluoConfiguration] object with all necessary properties which are all +properties with the `fluo.client.*` prefix in [fluo.properties]: + +```java +FluoConfiguration config = new FluoConfiguration(new File("fluo.properties")); +``` + +You can also create an empty [FluoConfiguration] object and set properties using Java: + +```java +FluoConfiguration config = new FluoConfiguration(); +config.setAccumuloUser("user"); +config.setAccumuloPassword("pass"); +config.setAccumuloInstance("instance"); +``` + +Once you have [FluoConfiguration] object, pass it to the `newClient()` method of [FluoFactory] to +create a [FluoClient]: + +```java +FluoClient client = FluoFactory.newClient(config) +``` + +It may help to reference the [API javadocs][API] while you are learning the Fluo API. + +## Running application code + +The `fluo exec <app name> <class> {arguments}` provides an easy way to execute application code. It +will execute a class with a main method if a jar containing the class is placed in the lib directory +of the application. When the class is run, Fluo classes and dependencies will be on the classpath. +The `fluo exec` command can inject the applications configuration if the class is written in the +following way. Defining the injection point is optional. + +```java +import javax.inject.Inject; + +public class AppCommand { + + //when run with fluo exec command, the applications configuration will be injected + @Inject + private static FluoConfiguration fluoConfig; + + public static void main(String[] args) throws Exception { + try(FluoClient fluoClient = FluoFactory.newClient(fluoConfig)) { + //do stuff with Fluo + } + } +} +``` + +## Creating a Fluo observer + +To create an observer, follow these steps: + +1. Create a class that extends [AbstractObserver]. +2. Build a jar containing this class and include this jar in the `lib/` directory of your Fluo + application. +3. Configure your Fluo instance to use this observer by modifying the Observer section of + [fluo.properties]. +4. Restart your Fluo instance so that your Fluo workers load the new observer. + +## Application Configuration + +Each observer can have its own configuration. This is useful for the case of using the same +observer code w/ different parameters. However for the case of sharing the same configuration +across observers, fluo provides a simple mechanism to set and access application specific +configuration. See the javadoc on [FluoClient].getAppConfiguration() for more details. + +## Debugging Applications + +While monitoring [Fluo metrics][metrics] can detect problems (like too many transaction collisions) +in a Fluo application, [metrics][metrics] may not provide enough information to debug the root cause +of the problem. To help debug Fluo applications, low-level logging of transactions can be turned on +by setting the following loggers to TRACE: + +| Logger | Level | Information | +|----------------------|-------|----------------------------------------------------------------------------------------------------| +| `fluo.tx` | TRACE | Provides detailed information about what transactions read and wrote | +| `fluo.tx.summary` | TRACE | Provides a one line summary about each transaction executed | +| `fluo.tx.collisions` | TRACE | Provides details about what data was involved When a transaction collides with another transaction | + +Below is an example log after setting `fluo.tx` to TRACE. The number following `txid: ` is the +transactions start timestamp from the Oracle. + +``` +2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 3 begin() thread: 198 +2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 3 class: com.SimpleLoader +2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 get(4333, stat count ) -> null +2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 3 set(4333, stat count , 1) +2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 3 commit() -> SUCCESSFUL commitTs: 4 +2015-02-11 18:24:05,341 [fluo.tx ] TRACE: txid: 5 begin() thread: 198 +2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 3 close() +2015-02-11 18:24:05,343 [fluo.tx ] TRACE: txid: 5 class: com.SimpleLoader +2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 get(4333, stat count ) -> 1 +2015-02-11 18:24:05,357 [fluo.tx ] TRACE: txid: 5 set(4333, stat count , 2) +2015-02-11 18:24:05,441 [fluo.tx ] TRACE: txid: 5 commit() -> SUCCESSFUL commitTs: 6 +2015-02-11 18:24:05,442 [fluo.tx ] TRACE: txid: 5 close() +``` + +The log above traces the following sequence of events. + +* Transaction T1 has a start timestamp of `3` +* Thread with id `198` is executing T1, its running code from the class `com.SimpleLoader` +* T1 reads row `4333` and column `stat count` which does not exist +* T1 sets row `4333` and column `stat count` to `1` +* T1 commits successfully and its commit timestamp from the Oracle is `4`. +* Transaction T2 has a start timestamp of `5` (because its `5` > `4` it can see what T1 wrote). +* T2 reads a value of `1` for row `4333` and column `stat count` +* T2 sets row `4333` and `column `stat count` to `2` +* T2 commits successfully with a commit timestamp of `6` + +Below is an example log after only setting `fluo.tx.collisions` to TRACE. This setting will only log +trace information when a collision occurs. Unlike the previous example, what the transaction read +and wrote is not logged. This shows that a transaction with a start timestamp of `106` and a class +name of `com.SimpleLoader` collided with another transaction on row `r1` and column `fam1 qual1`. + +``` +2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 class: com.SimpleLoader +2015-02-11 18:17:02,639 [tx.collisions] TRACE: txid: 106 collisions: {r1=[fam1 qual1 ]} +``` + +When applications read and write arbitrary binary data, this does not log so well. In order to make +the trace logs human readable, non ASCII chars are escaped using hex. The convention used it `\xDD` +where D is a hex digit. Also the `\` character is escaped to make the output unambiguous. + +[FluoFactory]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/client/FluoFactory.html +[FluoClient]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/client/FluoClient.html +[FluoConfiguration]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/config/FluoConfiguration.html +[AbstractObserver]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/observer/AbstractObserver.html +[fluo.properties]: https://github.com/apache/fluo/blob/rel/fluo-1.0.0-incubating/modules/distribution/src/main/config/fluo.properties +[API]: {{ site.fluo_api_base }}/1.0.0-incubating/ +[metrics]: /docs/fluo/1.0.0-incubating/metrics/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/architecture.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/architecture.md b/docs/fluo/1.0.0-incubating/architecture.md new file mode 100644 index 0000000..1817718 --- /dev/null +++ b/docs/fluo/1.0.0-incubating/architecture.md @@ -0,0 +1,45 @@ +--- +layout: fluo-doc +title: Fluo Architecture +version: 1.0.0-incubating +--- + +![fluo-architecture][1] + +## Fluo Application + +A **Fluo application** maintains a large scale computation using a series of small transactional +updates. Fluo applications store their data in a **Fluo table** which has a similar structure (row, +column, value) to an **Accumulo table** except that a Fluo table has no timestamps. A Fluo table +is implemented using an Accumulo table. While you could scan the Accumulo table used to implement +a Fluo table using an Accumulo client, you would read extra implementation-related data in addition +to your data. Therefore, developers should only interact with the data in a Fluo table by writing +Fluo client or observer code: + +* **Clients** ingest data or interact with Fluo from external applications (REST services, + crawlers, etc). +* **Observers** are run by Fluo workers and trigger a transaction when a requested column is + modified in the Fluo table. + +Multiple Fluo applications can run on a cluster at the same time. Each Fluo application runs as a +Hadoop YARN application and can be stopped, started, and upgraded independently. Fluo applications +consist of an oracle process and a configurable number of worker processes: + +* The **Oracle** process allocates timestamps for transactions. While only one Oracle is required, + Fluo can be configured to run extra Oracles that can take over if the primary Oracle fails. +* **Worker** processes run user code (called **observers**) that perform transactions. All workers + run the same observers. The number of worker instances are configured to handle the processing + workload. + +## Fluo Dependencies + +Fluo requires the following software to be running on the cluster: + +* **Accumulo** - Fluo stores its data in Accumulo and uses Accumulo's conditional mutations for + transactions. +* **Hadoop** - Each Fluo application run its oracle and worker processes as Hadoop YARN + applications. HDFS is also required for Accumulo. +* **Zookeeper** - Fluo stores its metadata and state information in Zookeeper. Zookeeper is also + required for Accumulo. + +[1]: /docs/fluo/1.0.0-incubating/resources/fluo-architecture.png http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/contributing.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/contributing.md b/docs/fluo/1.0.0-incubating/contributing.md new file mode 100644 index 0000000..4df5007 --- /dev/null +++ b/docs/fluo/1.0.0-incubating/contributing.md @@ -0,0 +1,31 @@ +--- +layout: fluo-doc +title: Contributing to Fluo +version: 1.0.0-incubating +--- + +## Building Fluo + +If you have [Git], [Maven], and [Java][java] (version 8+) installed, run these commands to build +Fluo: + + git clone https://github.com/apache/incubator-fluo.git + cd fluo + mvn package + +## Testing Fluo + +Fluo has a test suite that consists of the following: + +* Units tests which are run by `mvn package` +* Integration tests which are run using `mvn verify`. These tests start a local Fluo instance + (called MiniFluo) and run against it. + +## See Also + +* [How to Contribute][contribute] on Apache Fluo project website + +[Git]: http://git-scm.com/ +[java]: http://openjdk.java.net/ +[Maven]: http://maven.apache.org/ +[contribute]: https://fluo.apache.org/how-to-contribute/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/grafana.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/grafana.md b/docs/fluo/1.0.0-incubating/grafana.md new file mode 100644 index 0000000..c4a6d4c --- /dev/null +++ b/docs/fluo/1.0.0-incubating/grafana.md @@ -0,0 +1,84 @@ +--- +layout: fluo-doc +title: Fluo metrics in Grafana/InfluxDB +version: 1.0.0-incubating +--- + +This document describes how to send Fluo metrics to [InfluxDB], a time series database, and make +them viewable in [Grafana], a visualization tool. If you want general information on metrics, see +the [Fluo metrics][2] documentation. + +## Set up Grafana/InfluxDB on your own + +Follow the instructions below to setup InfluxDB and Grafana. + +1. Follow the standard installation instructions for [InfluxDB] and [Grafana]. As for versions, + the instructions below were written using InfluxDB v0.9.4.2 and Grafana v2.5.0. + +2. Add the following to your InfluxDB configuration to configure it accept metrics in Graphite + format from Fluo. The configuration below contains templates that transform the Graphite + metrics into a format that is usable in InfluxDB. + + ``` + [[graphite]] + bind-address = ":2003" + enabled = true + database = "fluo_metrics" + protocol = "tcp" + consistency-level = "one" + separator = "_" + batch-size = 1000 + batch-pending = 5 + batch-timeout = "1s" + templates = [ + "fluo.class.*.*.*.*.* ..app.host.measurement.observer.field", + "fluo.class.*.*.*.* ..app.host.measurement.observer", + "fluo.system.*.*.*.* ..app.host.measurement.field", + "fluo.system.*.*.* ..app.host.measurement", + "fluo.app.*.*.* ..host.measurement.field", + "fluo.app.*.* ..host.measurement", + ] + ``` + +3. Fluo distributes a file called `fluo_metrics_setup.txt` that contains a list of commands that + setup InfluxDB. These commands will configure an InfluxDB user, retention policies, and + continuous queries that downsample data for the historical dashboard in Grafana. Run the command + below to execute the commands in this file: + + ``` + $INFLUXDB_HOME/bin/influx -import -path $FLUO_HOME/contrib/influxdb/fluo_metrics_setup.txt + ``` + +3. Configure `fluo.properties` in your Fluo app configuration to send Graphite metrics to InfluxDB. + Below is example configuration. Remember to replace `<INFLUXDB_HOST>` with the actual host. + + ``` + fluo.metrics.reporter.graphite.enable=true + fluo.metrics.reporter.graphite.host=<INFLUXDB_HOST> + fluo.metrics.reporter.graphite.port=2003 + fluo.metrics.reporter.graphite.frequency=30 + ``` + + The reporting frequency of 30 sec is required if you are using the provided Grafana dashboards + that are configured in the next step. + +4. Grafana needs to be configured to load dashboard JSON templates from a directory. Fluo + distributes two Grafana dashboard templates in its tarball distribution in the directory + `contrib/grafana`. Before restarting Grafana, you should copy the templates from your Fluo + installation to the `dashboards/` directory configured below. + + ``` + [dashboards.json] + enabled = true + path = <GRAFANA_HOME>/dashboards + ``` + +5. If you restart Grafana, you will see the Fluo dashboards configured but all of their charts will + be empty unless you have a Fluo application running and configured to send data to InfluxDB. + When you start sending data, you may need to refresh the dashboard page in the browser to start + viewing metrics. + +[1]: https://dropwizard.github.io/metrics/3.1.0/ +[2]: /docs/fluo/1.0.0-incubating/metrics/ +[Grafana]: http://grafana.org/ +[InfluxDB]: https://influxdb.com/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/index.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/index.md b/docs/fluo/1.0.0-incubating/index.md new file mode 100644 index 0000000..4373f85 --- /dev/null +++ b/docs/fluo/1.0.0-incubating/index.md @@ -0,0 +1,46 @@ +--- +layout: fluo-doc +title: Fluo 1.0.0-incubating Documentation +version: 1.0.0-incubating +--- + +**Apache Fluo lets users make incremental updates to large data sets stored in Apache Accumulo.** + +Apache Fluo is an open source implementation of [Percolator][percolator] (which populates +Google's search index) for [Apache Accumulo][accumulo]. Fluo makes it possible to update the results +of a large-scale computation, index, or analytic as new data is discovered. + +## Getting Started + +* Take the [Fluo Tour][tour] if you are completely new to Fluo. +* Read the [install instructions][install] to install Fluo and start a Fluo application in YARN on a + cluster where Accumulo, Hadoop & Zookeeper are running. If you need help setting up these + dependencies, see the [related projects page][related] for external projects that may help. + +## Applications + +Below are helpful resources for Fluo application developers: + +* [Instructions][apps] for creating Fluo applications +* [Fluo API][api] javadocs +* [Fluo Recipes][recipes] is a project that provides common code for Fluo application developers + implemented using the Fluo API. + +## Implementation + +* [Architecture] - Overview of Fluo's architecture +* [Contributing] - Documentation for developers who want to contribute to Fluo +* [Metrics] - Fluo metrics are visible via JMX by default but can be configured to send to Graphite + or Ganglia + +[related]: /related-projects/ +[tour]: /tour/ +[accumulo]: https://accumulo.apache.org +[percolator]: https://research.google.com/pubs/pub36726.html +[install]: /docs/fluo/1.0.0-incubating/install/ +[apps]: /docs/fluo/1.0.0-incubating/applications/ +[api]: {{ site.fluo_api_base }}/1.0.0-incubating/ +[recipes]: https://github.com/apache/incubator-fluo-recipes +[Metrics]: /docs/fluo/1.0.0-incubating/metrics/ +[Contributing]: /docs/fluo/1.0.0-incubating/contributing/ +[Architecture]: /docs/fluo/1.0.0-incubating/architecture/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/install.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/install.md b/docs/fluo/1.0.0-incubating/install.md new file mode 100644 index 0000000..a897529 --- /dev/null +++ b/docs/fluo/1.0.0-incubating/install.md @@ -0,0 +1,210 @@ +--- +layout: fluo-doc +title: Fluo Install Instructions +version: 1.0.0-incubating +--- + +Instructions for installing Apache Fluo and starting a Fluo application in YARN on a cluster where +Accumulo, Hadoop & Zookeeper are running. If you need help setting up these dependencies, see the +[related projects page][related] for external projects that may help. + +## Requirements + +Before you install Fluo, the following software must be installed and running on your local machine +or cluster: + +| Software | Recommended Version | Minimum Version | +|-------------|---------------------|-----------------| +| [Accumulo] | 1.7.2 | 1.6.1 | +| [Hadoop] | 2.7.2 | 2.6.0 | +| [Zookeeper] | 3.4.8 | | +| [Java] | JDK 8 | JDK 8 | + +## Obtain a distribution + +Before you can install Fluo, you will need to obtain a distribution tarball. It is recommended that +you download the [latest release][release]. You can also build a distribution from the master +branch by following these steps which create a tarball in `modules/distribution/target`: + + git clone https://github.com/apache/incubator-fluo.git + cd fluo/ + mvn package + +## Install Fluo + +After you obtain a Fluo distribution tarball, follow these steps to install Fluo. + +1. Choose a directory with plenty of space and untar the distribution: + + tar -xvzf fluo-1.0.0-incubating-bin.tar.gz + +2. Copy the example configuration to the base of your configuration directory to create the default + configuration for your Fluo install: + + cp conf/examples/* conf/ + + The default configuration will be used as the base configuration for each new application. + +3. Modify [fluo.properties] for your environment. However, you should not configure any + application settings (like observers). + + NOTE - All properties that have a default are set with it. Uncomment a property if you want + to use a value different than the default. Properties that are unset and uncommented must be + set by the user. + +4. Fluo needs to build its classpath using jars from the versions of Hadoop, Accumulo, and +Zookeeper that you are using. Choose one of the two ways below to make these jars available +to Fluo: + + * Set `HADOOP_PREFIX`, `ACCUMULO_HOME`, and `ZOOKEEPER_HOME` in your environment or configure + these variables in [fluo-env.sh]. Fluo will look in these locations for jars. + * Run `./lib/fetch.sh ahz` to download Hadoop, Accumulo, and Zookeeper jars to `lib/ahz` and + configure [fluo-env.sh] to look in this directory. By default, this command will download the + default versions set in [lib/ahz/pom.xml]. If you are not using the default versions, you can + override them: + + ./lib/fetch.sh ahz -Daccumulo.version=1.7.2 -Dhadoop.version=2.7.2 -Dzookeeper.version=3.4.8 + +5. Fluo needs more dependencies than what is available from Hadoop, Accumulo, and Zookeeper. These + extra dependencies need to be downloaded to `lib/` using the command below: + + ./lib/fetch.sh extra + +You are now ready to use the Fluo command script. + +## Fluo command script + +The Fluo command script is located at `bin/fluo` of your Fluo installation. All Fluo commands are +invoked by this script. + +Modify and add the following to your `~/.bashrc` if you want to be able to execute the fluo script +from any directory: + + export PATH=/path/to/fluo-1.0.0-incubating/bin:$PATH + +Source your `.bashrc` for the changes to take effect and test the script + + source ~/.bashrc + fluo + +Running the script without any arguments prints a description of all commands. + + ./bin/fluo + +## Configure a Fluo application + +You are now ready to configure a Fluo application. Use the command below to create the +configuration necessary for a new application. Feel free to pick a different name (other than +`myapp`) for your application: + + fluo new myapp + +This command will create a directory for your application at `apps/myapp` of your Fluo install which +will contain a `conf` and `lib`. + +The `apps/myapp/conf` directory contains a copy of the `fluo.properties` from your default +configuration. This should be configured for your application: + + vim apps/myapp/fluo.properties + +When configuring the observer section in fluo.properties, you can configure your instance for the +[phrasecount] application if you have not created your own application. See the [phrasecount] +example for instructions. You can also choose not to configure any observers but your workers will +be idle when started. + +The `apps/myapp/lib` directory should contain any observer jars for your application. If you +configured [fluo.properties] for observers, copy any jars containing these observer classes to this +directory. + +## Initialize your application + +After your application has been configured, use the command below to initialize it: + + fluo init myapp + +This only needs to be called once and stores configuration in Zookeeper. + +## Start your application + +A Fluo application consists of one oracle process and multiple worker processes. Before starting +your application, you can configure the number of worker process in your [fluo.properties] file. + +When you are ready to start your Fluo application on your YARN cluster, run the command below: + + fluo start myapp + +The start command above will work for a single-node or a large cluster. By using YARN, you do not +need to deploy the Fluo binaries to every node on your cluster or start processes on every node. + +You can use the following command to check the status of your instance: + + fluo status myapp + +For more detailed information on the YARN containers running Fluo: + + fluo info myapp + +You can also use `yarn application -list` to check the status of your Fluo instance in YARN. Logs +are viewable within YARN. + +When you have data in your fluo instance, you can view it using the command `fluo scan`. Pipe the +output to `less` using the command `fluo scan | less` if you want to page through the data. + +## Stop your Fluo application + +Use the following command to stop your Fluo application: + + fluo stop myapp + +If stop fails, there is also a kill command. + + fluo kill myapp + +## Tuning Accumulo + +Fluo will reread the same data frequently when it checks conditions on mutations. When Fluo +initializes a table it enables data caching to make this more efficient. However you may need to +increase the amount of memory available for caching in the tserver by increasing +`tserver.cache.data.size`. Increasing this may require increasing the maximum tserver java heap size +in `accumulo-env.sh`. + +Fluo will run many client threads, will want to ensure the tablet server has enough threads. Should +probably increase the `tserver.server.threads.minimum` Accumulo setting. + +Using at least Accumulo 1.6.1 is recommended because multiple performance bugs were fixed. + +## Tuning YARN + +When running Fluo oracles and workers in YARN, the number of instances, max memory, and number of +cores for Fluo processes can be configured in [fluo.properties]. If YARN is killing processes +consider increasing `twill.java.reserved.memory.mb` (which defaults to 200 and is set in +yarn-site.xml). The `twill.java.reserved.memory.mb` config determines the gap between the YARN +memory limit set in [fluo.properties] and the java -Xmx setting. For example, if max memory is 1024 +and twill reserved memory is 200, the java -Xmx setting will be 1024-200 = 824 MB. + +## Run locally without YARN + +If you do not have YARN set up, you can start the oracle and worker as a local Fluo process using +the following commands: + + local-fluo start-oracle + local-fluo start-worker + +Use the following commands to stop a local Fluo process: + + local-fluo stop-worker + local-fluo stop-oracle + +In a distributed environment, you will need to deploy and configure a Fluo distribution on every +node in your cluster. + +[Accumulo]: https://accumulo.apache.org/ +[Hadoop]: http://hadoop.apache.org/ +[Zookeeper]: http://zookeeper.apache.org/ +[Java]: http://openjdk.java.net/ +[related]: https://fluo.apache.org/related-projects/ +[release]: https://fluo.apache.org/download/ +[phrasecount]: https://github.com/fluo-io/phrasecount +[fluo.properties]: https://github.com/apache/fluo/blob/rel/fluo-1.0.0-incubating/modules/distribution/src/main/config/fluo.properties +[fluo-env.sh]: https://github.com/apache/fluo/blob/rel/fluo-1.0.0-incubating/modules/distribution/src/main/config/fluo-env.sh +[lib/ahz/pom.xml]: https://github.com/apache/fluo/blob/rel/fluo-1.0.0-incubating/modules/distribution/src/main/lib/ahz/pom.xml http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/metrics.md ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/metrics.md b/docs/fluo/1.0.0-incubating/metrics.md new file mode 100644 index 0000000..e4d9bf2 --- /dev/null +++ b/docs/fluo/1.0.0-incubating/metrics.md @@ -0,0 +1,117 @@ +--- +layout: fluo-doc +title: Fluo Metrics +version: 1.0.0-incubating +--- + +A Fluo application can be configured (in [fluo.properties]) to report metrics. When metrics are +configured, Fluo will report some 'default' metrics about an application that help users monitor its +performance. Users can also write code to report 'application-specific' metrics from their +applications. Both 'application-specific' and 'default' metrics share the same reporter configured +by [fluo.properties] and are described in detail below. + +## Configuring reporters + +Fluo metrics are not published by default. To publish metrics, configure a reporter in the 'metrics' +section of [fluo.properties]. There are several different reporter types (i.e Console, CSV, +Graphite, JMX, SLF4J) that are implemented using [Dropwizard]. The choice of which reporter to use +depends on the visualization tool used. If you are not currently using a visualization tool, there +is [documentation][grafana] for reporting Fluo metrics to Grafana/InfluxDB. + +## Metrics names + +When Fluo metrics are reported, they are published using a naming scheme that encodes additional +information. This additional information is represented using all caps variables (i.e `METRIC`) +below. + +Default metrics start with `fluo.class` or `fluo.system` and have following naming schemes: + + fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS + fluo.system.APPLICATION.REPORTER_ID.METRIC + +Application metrics start with `fluo.app` and have following scheme: + + fluo.app.REPORTER_ID.METRIC + +The variables below describe the additional information that is encoded in metrics names. + +1. `APPLICATION` - Fluo application name +2. `REPORTER_ID` - Unique ID of the Fluo oracle, worker, or client that is reporting the metric. + When running in YARN, this ID is of the format `worker-INSTANCE_ID` or `oracle-INSTANCE_ID` + where `INSTANCE_ID` corresponds to instance number. When not running in YARN, this ID consists + of a hostname and a base36 long that is unique across all fluo processes. +3. `METRIC` - Name of the metric. For 'default' metrics, this is set by Fluo. For 'application' + metrics, this is set by user. Name should be unique and avoid using period '.' in name. +4. `CLASS` - Name of Fluo observer or loader class that produced metric. This allows things like + transaction collisions to be tracked per class. + +## Application-specific metrics + +Application metrics are implemented by retrieving a [MetricsReporter] from an [Observer], [Loader], +or [FluoClient]. These metrics are named using the format `fluo.app.REPORTER_ID.METRIC`. + +## Default metrics + +Default metrics report for a particular Observer/Loader class or system-wide. + +Below are metrics that are reported from each Observer/Loader class that is configured in a Fluo +application. These metrics are reported after each transaction and named using the format +`fluo.class.APPLICATION.REPORTER_ID.METRIC.CLASS`. + +* tx_lock_wait_time - [Timer] + - Time transaction spent waiting on locks held by other transactions. + - Only updated for transactions that have non-zero lock time. +* tx_execution_time - [Timer] + - Time transaction took to execute. + - Updated for failed and successful transactions. + - This does not include commit time, only the time from start until commit is called. +* tx_with_collision - [Meter] + - Rate of transactions with collisions. +* tx_collisions - [Meter] + - Rate of collisions. +* tx_entries_set - [Meter] + - Rate of row/columns set by transaction +* tx_entries_read - [Meter] + - Rate of row/columns read by transaction that existed. + - There is currently no count of all reads (including non-existent data) +* tx_locks_timedout - [Meter] + - Rate of timedout locks rolled back by transaction. + - These are locks that are held for very long periods by another transaction that appears to be + alive based on zookeeper. +* tx_locks_dead - [Meter] + - Rate of dead locks rolled by a transaction. + - These are locks held by a process that appears to be dead according to zookeeper. +* tx_status_`<STATUS>` - [Meter] + - Rate of different ways (i.e `<STATUS>`) a transaction can terminate + +Below are system-wide metrics that are reported for the entire Fluo application. These metrics are +named using the format `fluo.system.APPLICATION.REPORTER_ID.METRIC`. + +* oracle_response_time - [Timer] + - Time each RPC call to oracle for stamps took +* oracle_client_stamps - [Histogram] + - Number of stamps requested for each request for stamps to the server +* oracle_server_stamps - [Histogram] + - Number of stamps requested for each request for stamps from a client +* worker_notifications_queued - [Gauge] + - The current number of notifications queued for processing. +* transactor_committing - [Gauge] + - The current number of transactions that are working their way through the commit steps. + +Histograms and Timers have a counter. In the case of a histogram, the counter is the number of times +the metric was updated and not a sum of the updates. For example if a request for 5 timestamps was +made to the oracle followed by a request for 3 timestamps, then the count for `oracle_server_stamps` +would be 2 and the mean would be (5+3)/2. + +[fluo.properties]: https://github.com/apache/fluo/blob/rel/fluo-1.0.0-incubating/modules/distribution/src/main/config/fluo.properties +[Dropwizard]: https://dropwizard.github.io/metrics/3.1.0/ +[grafana]: /docs/fluo/1.0.0-incubating/grafana/ +[MetricsReporter]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/metrics/MetricsReporter.html +[Observer]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/observer/Observer.html +[Loader]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/client/Loader.html +[FluoClient]: {{ site.fluo_api_static }}/1.0.0-incubating/org/apache/fluo/api/client/FluoClient.html +[Timer]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#timers +[Counter]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#counters +[Histogram]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#histograms +[Gauge]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#gauges +[Meter]: https://dropwizard.github.io/metrics/3.1.0/getting-started/#meters http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/1.0.0-incubating/resources/fluo-architecture.png ---------------------------------------------------------------------- diff --git a/docs/fluo/1.0.0-incubating/resources/fluo-architecture.png b/docs/fluo/1.0.0-incubating/resources/fluo-architecture.png new file mode 100644 index 0000000..3ba96fd Binary files /dev/null and b/docs/fluo/1.0.0-incubating/resources/fluo-architecture.png differ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/fluo/index.md ---------------------------------------------------------------------- diff --git a/docs/fluo/index.md b/docs/fluo/index.md deleted file mode 100644 index 019d87d..0000000 --- a/docs/fluo/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -layout: page -title: Fluo Documentation Archive ---- - -Fluo documentation before Apache incubation: - -* [1.0.0-beta-2][fluo-b2] - January 12, 2016 -* [1.0.0-beta-1][fluo-b1] - June 9, 2015 -* [1.0.0-alpha-1][fluo-a1] - October 2, 2014 - -[fluo-b2]: /docs/fluo/1.0.0-beta-2/ -[fluo-b1]: /docs/fluo/1.0.0-beta-1/ -[fluo-a1]: /docs/fluo/1.0.0-alpha-1/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/index.md ---------------------------------------------------------------------- diff --git a/docs/index.md b/docs/index.md index 5298cc9..961d401 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,41 +1,25 @@ --- layout: page title: Documentation +redirect_from: + - /docs/fluo/ + - /docs/fluo-recipes/ --- For a general overview of Fluo, take the [Fluo tour](/tour/). [Apache Fluo] and [Apache Fluo Recipes] have seperate documentation as they are different repositories with their own release cycle. -## Apache Fluo +### Apache Fluo documentation -Starting with 1.0.0-incubating, Apache Fluo will follow [semver](http://semver.org/) for all API -changes. The API consist of evertyhing under the org.apache.fluo.api package. Code outside of this -package can change at any time. If your project is using Fluo code that falls outside of the API, -then consider [initiating a discussion](/getinvolved/) about adding it to the API. +* [1.0.0-incubating][fluo-1.0] - September 27, 2016 -Last release was `{{ site.latest_fluo_release }}` made on {{ site.latest_fluo_release_date }}. +### Apache Fluo Recipes documentation -* Documentation - [Latest][fluo-docs-latest] \| [Archive][fluo-docs-archive] -* API - [Latest][fluo-api-latest] \| [Archive][fluo-api-archive] -* Release summary - [Latest][fluo-sum-latest] \| [Archive][fluo-sum-archive] -* Release notes - [Latest][fluo-notes-latest] \| [Archive][fluo-notes-archive] +* The first release of Apache Fluo Recipes has not been made yet. -## Apache Fluo Recipes +Documentation for releases before joining Apache are available in the [Pre-ASF archive](archive). -The first release of Apache Fluo Recipes has not been made yet. - -## Pre Apache Documentation - -Documentation about releases made before Apache Fluo entered incubation at Apache has been moved [here](pre-asf) - -[fluo-docs-latest]: /docs/fluo/{{ site.latest_fluo_release }}/ -[fluo-api-latest]: /apidocs/fluo/{{ site.latest_fluo_release }}/ -[fluo-sum-latest]: /release-summaries/{{ site.latest_fluo_release }}/ -[fluo-notes-latest]: /release-notes/{{ site.latest_fluo_release }}/ [Apache Fluo]: https://github.com/apache/fluo [Apache Fluo Recipes]: https://github.com/apache/fluo-recipes -[fluo-docs-archive]: /docs/fluo/ -[fluo-api-archive]: /apidocs/fluo/ -[fluo-sum-archive]: /release-summaries/ -[fluo-notes-archive]: /release-notes/ +[fluo-1.0]: /docs/fluo/1.0.0-incubating/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/docs/pre-asf.md ---------------------------------------------------------------------- diff --git a/docs/pre-asf.md b/docs/pre-asf.md deleted file mode 100644 index 4018bd7..0000000 --- a/docs/pre-asf.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -layout: page -title: Pre Apache Documentation ---- - -<div class="alert alert-danger" role="alert">Please note - This is documentation related to non-ASF releases of Fluo. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> - -## Fluo - -Last non-ASF release was `1.0.0-beta-2` made on January 12, 2016. - -* Documentation - [Latest][fluo-docs-latest] \| [Archive][fluo-docs-archive] -* API - [Latest][fluo-api-latest] \| [Archive][fluo-api-archive] -* Release summary - [Latest][fluo-sum-latest] \| [Archive][fluo-sum-archive] -* Release notes - [Latest][fluo-notes-latest] \| [Archive][fluo-notes-archive] - -## Fluo Recipes - -Last non-ASF release was `1.0.0-beta-2` made on March 29, 2016. - -* Documentation - [Latest][recipes-docs-latest] \| [Archive][recipes-docs-archive] -* API - [Latest][recipes-api-latest] \| [Archive][recipes-api-archive] - -[fluo-docs-latest]: /docs/fluo/1.0.0-beta-2/ -[fluo-api-latest]: /apidocs/fluo/1.0.0-beta-2/ -[fluo-sum-latest]: /release-summaries/1.0.0-beta-2/ -[fluo-notes-latest]: /release-notes/1.0.0-beta-2/ -[recipes-docs-latest]: /docs/fluo-recipes/1.0.0-beta-2/ -[recipes-api-latest]: /apidocs/fluo-recipes/1.0.0-beta-2/ -[fluo-docs-archive]: /docs/fluo/ -[fluo-api-archive]: /apidocs/fluo/ -[fluo-sum-archive]: /release-summaries/ -[fluo-notes-archive]: /release-notes/ -[recipes-docs-archive]: /docs/fluo-recipes/ -[recipes-api-archive]: /apidocs/fluo-recipes/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/index.html ---------------------------------------------------------------------- diff --git a/index.html b/index.html index f8e8ae4..730f947 100644 --- a/index.html +++ b/index.html @@ -56,7 +56,7 @@ html_title_override: true </div> <div class="col-sm-4"> <h4>Core API</h4> - <p>The core <a href="{{ site.baseurl }}/apidocs/fluo/{{ site.latest_fluo_release }}/">Fluo API</a> supports simple transactional updates using get/set methods.</li> + <p>The core <a href="{{ site.baseurl }}/docs/fluo/{{ site.latest_fluo_release }}/">Fluo API</a> supports simple transactional updates using get/set methods.</li> </div> </div> http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/pages/people.md ---------------------------------------------------------------------- diff --git a/pages/people.md b/pages/people.md index f217427..4e7b2c9 100644 --- a/pages/people.md +++ b/pages/people.md @@ -4,9 +4,6 @@ title: People permalink: /people/ --- -Github also has a [contributor list](https://github.com/apache/incubator-fluo/graphs/contributors) -based on commits. - --- ## PPMC/Committers @@ -31,6 +28,9 @@ based on commits. ## Contributors +GitHub also has a [contributor list](https://github.com/apache/incubator-fluo/graphs/contributors) +based on commits. + | Name | Organization | Timezone | ------------------------------------------------------|-------------------------------------|------------| | [Eric Newton](https://github.com/ericnewton) | | [ET][et] | http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/pages/pre-asf-release.md ---------------------------------------------------------------------- diff --git a/pages/pre-asf-release.md b/pages/pre-asf-release.md new file mode 100644 index 0000000..cd7785b --- /dev/null +++ b/pages/pre-asf-release.md @@ -0,0 +1,15 @@ +--- +layout: page +title: Pre-ASF releases +permalink: "/pre-asf-release/" +--- + +<div class="alert alert-danger" role="alert">Please note - This page links to non-ASF releases. These releases were made before Apache Fluo became an ASF project and are not endorsed by the ASF.</div> + +{% for release in site.categories.release %} +{% if release.historical %} +* [{{ release.version }}]({{ site.baseurl }}/release/{{ release.version }}/) - {{ release.date | date_to_string }} +{% endif %} +{% endfor %} + + http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/pages/release-notes-archive.md ---------------------------------------------------------------------- diff --git a/pages/release-notes-archive.md b/pages/release-notes-archive.md deleted file mode 100644 index c0f959b..0000000 --- a/pages/release-notes-archive.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -layout: page -title: Release Notes Archive -permalink: "/release-notes/" ---- - -Apache Fluo release notes: - -{% for release in site.categories.release-notes %} -{% unless release.historical %} -* [{{ release.version }}]({{ site.baseurl }}/release-notes/{{ release.version }}/) - {{ release.date | date_to_string }} -{% endunless %} -{% endfor %} - -Fluo release notes before Apache incubation: - -{% for release in site.categories.release-notes %} -{% if release.historical %} -* [{{ release.version }}]({{ site.baseurl }}/release-notes/{{ release.version }}/) - {{ release.date | date_to_string }} -{% endif %} -{% endfor %} http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/pages/release-notes.md ---------------------------------------------------------------------- diff --git a/pages/release-notes.md b/pages/release-notes.md new file mode 100644 index 0000000..8a2346c --- /dev/null +++ b/pages/release-notes.md @@ -0,0 +1,21 @@ +--- +layout: page +title: Release Notes +permalink: "/release-notes/" +--- + +#### Apache Fluo release notes + +{% for release in site.categories.release-notes %} +{% unless release.historical %} +* [{{ release.version }}]({{ site.baseurl }}/release-notes/{{ release.version }}/) - {{ release.date | date_to_string }} +{% endunless %} +{% endfor %} + +#### Release notes before Apache + +{% for release in site.categories.release-notes %} +{% if release.historical %} +* [{{ release.version }}]({{ site.baseurl }}/release-notes/{{ release.version }}/) - {{ release.date | date_to_string }} +{% endif %} +{% endfor %} http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/pages/release-summaries-archive.md ---------------------------------------------------------------------- diff --git a/pages/release-summaries-archive.md b/pages/release-summaries-archive.md deleted file mode 100644 index 4355d96..0000000 --- a/pages/release-summaries-archive.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -layout: page -title: Release Summaries Archive -permalink: "/release-summaries/" ---- - -Apache Fluo release summaries: - -{% for release in site.categories.release-summaries %} -{% unless release.historical %} -* [{{ release.version }}]({{ site.baseurl }}/release-summaries/{{ release.version }}/) - {{ release.date | date_to_string }} -{% endunless %} -{% endfor %} - -Fluo release summaries before Apache incubation: - -{% for release in site.categories.release-summaries %} -{% if release.historical %} -* [{{ release.version }}]({{ site.baseurl }}/release-summaries/{{ release.version }}/) - {{ release.date | date_to_string }} -{% endif %} -{% endfor %} - - http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/pages/release.md ---------------------------------------------------------------------- diff --git a/pages/release.md b/pages/release.md new file mode 100644 index 0000000..44f95ff --- /dev/null +++ b/pages/release.md @@ -0,0 +1,15 @@ +--- +layout: page +title: Releases +permalink: "/release/" +--- + +Apache Fluo and Apache Fluo Recipes are released seperately on their own schedule. + +{% for release in site.categories.release %} +{% unless release.historical %} +* [{{ release.version }}]({{ site.baseurl }}/release/{{ release.version }}/) - {{ release.date | date_to_string }} +{% endunless %} +{% endfor %} + +Releases before joining Apache are available in the [Pre-ASF release archive](/pre-asf-release/). http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/tour/application-configuration.md ---------------------------------------------------------------------- diff --git a/tour/application-configuration.md b/tour/application-configuration.md index 676d2f1..d02c33f 100644 --- a/tour/application-configuration.md +++ b/tour/application-configuration.md @@ -158,11 +158,11 @@ Exporting val:555 from row:e:99 to db/table:db1/table3 Exporting val:999 from row:e:98 to db/table:db1/table9 ``` -[fcogac]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/config/FluoConfiguration.html#getAppConfiguration-- -[fclgac]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/FluoClient.html#getAppConfiguration-- -[ocgac]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/observer/Observer.Context.html#getAppConfiguration-- -[lcgac]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/Loader.Context.html#getAppConfiguration-- -[ospec]: /apidocs/fluo/{{ site.latest_fluo_release}}/org/apache/fluo/api/config/ObserverSpecification.html -[ocgp]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/observer/Observer.Context.html#getObserverConfiguration-- +[fcogac]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/config/FluoConfiguration.html#getAppConfiguration-- +[fclgac]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/FluoClient.html#getAppConfiguration-- +[ocgac]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/observer/Observer.Context.html#getAppConfiguration-- +[lcgac]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/Loader.Context.html#getAppConfiguration-- +[ospec]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release}}/org/apache/fluo/api/config/ObserverSpecification.html +[ocgp]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/observer/Observer.Context.html#getObserverConfiguration-- http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/tour/basic-read-write.md ---------------------------------------------------------------------- diff --git a/tour/basic-read-write.md b/tour/basic-read-write.md index f244aec..fd64673 100644 --- a/tour/basic-read-write.md +++ b/tour/basic-read-write.md @@ -40,6 +40,6 @@ Transactions and snapshots allocate resources and therefore have `close()` metho try-with-resources block will automatically call `close()`, even when exceptions occur. [1]: https://github.com/apache/incubator-fluo-website/tree/fluo-tour -[get]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/SnapshotBase.html#get-org.apache.fluo.api.data.Bytes-org.apache.fluo.api.data.Column- -[gets]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/SnapshotBase.html#gets-java.lang.CharSequence-org.apache.fluo.api.data.Column- -[Bytes]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.html +[get]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/SnapshotBase.html#get-org.apache.fluo.api.data.Bytes-org.apache.fluo.api.data.Column- +[gets]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/client/SnapshotBase.html#gets-java.lang.CharSequence-org.apache.fluo.api.data.Column- +[Bytes]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.html http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/tour/data-pojos.md ---------------------------------------------------------------------- diff --git a/tour/data-pojos.md b/tour/data-pojos.md index 355355d..8b211f7 100644 --- a/tour/data-pojos.md +++ b/tour/data-pojos.md @@ -14,8 +14,8 @@ like String is an immutable wrapper around `char[]`. To make building [Bytes] more efficient and easier, there is a reusable [BytesBuilder] that is modeled after StringBuilder. Call [Bytes.builder()][nb] to create a new [BytesBuilder]. -[data-pkg]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/package-summary.html -[Bytes]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.html -[BytesBuilder]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.BytesBuilder.html -[nb]: /apidocs/fluo/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.html#builder-- +[data-pkg]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/package-summary.html +[Bytes]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.html +[BytesBuilder]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.BytesBuilder.html +[nb]: {{ site.fluo_api_static }}/{{ site.latest_fluo_release }}/org/apache/fluo/api/data/Bytes.html#builder--
