Refactored website for 1.0.0-incubating release * Set correct date for 1.0.0-incubating release * Release summary pages are now called releases * Added 'Releases', 'Tour', 'Docs', & 'API' links to main page * Removed download link on main page as each release has link * Release notes and release 'posts' can now be either Fluo or Fluo Recipes * API and docs pages each have index and archive to pre-ASF * Added documentation and blog post for 1.0.0-incubating release * Updated github-pages to version 97 * Updated links to external javadocs
Project: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/commit/0495f99c Tree: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/tree/0495f99c Diff: http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/diff/0495f99c Branch: refs/heads/gh-pages Commit: 0495f99c7ef2783f06a7e92ad273436b783fd23e Parents: 64e1b76 Author: Mike Walch <mwa...@apache.org> Authored: Fri Oct 7 14:00:06 2016 -0400 Committer: Mike Walch <mwa...@apache.org> Committed: Thu Oct 13 14:34:05 2016 -0400 ---------------------------------------------------------------------- Gemfile | 2 +- Gemfile.lock | 48 +++-- README.md | 24 +-- _config.yml | 8 +- _layouts/default.html | 6 +- _layouts/release-summary.html | 21 -- _layouts/release.html | 21 ++ ...2014-10-02-fluo-1.0.0-alpha-1-is-released.md | 6 +- .../2015-06-09-fluo-1.0.0-beta-1-is-released.md | 4 +- ...2015-12-22-beta-2-pre-release-stress-test.md | 2 +- .../2016-01-12-fluo-1.0.0-beta-2-is-released.md | 4 +- .../2016-09-27-1.0.0-incubating-is-released.md | 11 + .../release-notes/2014-10-02-1.0.0-alpha-1.md | 129 ------------ .../2014-10-02-fluo-1.0.0-alpha-1.md | 129 ++++++++++++ _posts/release-notes/2015-06-09-1.0.0-beta-1.md | 177 ---------------- .../2015-06-09-fluo-1.0.0-beta-1.md | 177 ++++++++++++++++ _posts/release-notes/2016-01-12-1.0.0-beta-2.md | 157 -------------- .../2016-01-12-fluo-1.0.0-beta-2.md | 157 ++++++++++++++ .../2016-09-27-fluo-1.0.0-incubating.md | 117 +++++++++++ .../2016-10-04-1.0.0-incubating.md | 112 ---------- .../2014-10-02-1.0.0-alpha-1.md | 90 -------- .../2015-06-09-1.0.0-beta-1.md | 35 ---- .../2016-01-12-1.0.0-beta-2.md | 40 ---- _posts/release/2014-10-02-fluo-1.0.0-alpha-1.md | 90 ++++++++ _posts/release/2015-06-09-fluo-1.0.0-beta-1.md | 35 ++++ _posts/release/2016-01-12-fluo-1.0.0-beta-2.md | 40 ++++ .../2016-01-13-fluo-recipes-1.0.0-beta-1.md | 19 ++ .../2016-03-29-fluo-recipes-1.0.0-beta-2.md | 19 ++ .../release/2016-09-27-fluo-1.0.0-incubating.md | 27 +++ _scripts/convert-docs.py | 6 + apidocs/archive.md | 23 ++ apidocs/fluo-recipes/index.md | 15 -- apidocs/fluo/archive.md | 16 -- apidocs/fluo/index.md | 13 -- apidocs/index.md | 19 +- docs/archive.md | 23 ++ docs/fluo-recipes/index.md | 12 -- docs/fluo/1.0.0-beta-1/contributing.md | 4 +- docs/fluo/1.0.0-beta-2/contributing.md | 4 +- docs/fluo/1.0.0-incubating/applications.md | 167 +++++++++++++++ docs/fluo/1.0.0-incubating/architecture.md | 45 ++++ docs/fluo/1.0.0-incubating/contributing.md | 31 +++ docs/fluo/1.0.0-incubating/grafana.md | 84 ++++++++ docs/fluo/1.0.0-incubating/index.md | 46 ++++ docs/fluo/1.0.0-incubating/install.md | 210 +++++++++++++++++++ docs/fluo/1.0.0-incubating/metrics.md | 117 +++++++++++ .../resources/fluo-architecture.png | Bin 0 -> 61085 bytes docs/fluo/index.md | 14 -- docs/index.md | 34 +-- docs/pre-asf.md | 35 ---- index.html | 2 +- pages/people.md | 6 +- pages/pre-asf-release.md | 15 ++ pages/release-notes-archive.md | 21 -- pages/release-notes.md | 21 ++ pages/release-summaries-archive.md | 23 -- pages/release.md | 15 ++ tour/application-configuration.md | 12 +- tour/basic-read-write.md | 6 +- tour/data-pojos.md | 8 +- tour/loader-executer.md | 10 +- tour/multi-get.md | 8 +- tour/scanning.md | 4 +- 63 files changed, 1750 insertions(+), 1026 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/Gemfile ---------------------------------------------------------------------- diff --git a/Gemfile b/Gemfile index 26663c8..429b32b 100644 --- a/Gemfile +++ b/Gemfile @@ -1,3 +1,3 @@ source 'https://rubygems.org' -gem 'github-pages', '89' +gem 'github-pages', '97' http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/Gemfile.lock ---------------------------------------------------------------------- diff --git a/Gemfile.lock b/Gemfile.lock index 9d8255a..23827bc 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -12,36 +12,39 @@ GEM coffee-script-source execjs coffee-script-source (1.10.0) - colorator (0.1) - ethon (0.9.0) + colorator (1.1.0) + ethon (0.9.1) ffi (>= 1.3.0) execjs (2.7.0) faraday (0.9.2) multipart-post (>= 1.2, < 3) ffi (1.9.14) + forwardable-extended (2.6.0) gemoji (2.1.0) - github-pages (89) + github-pages (97) activesupport (= 4.2.7) - github-pages-health-check (= 1.1.0) - jekyll (= 3.1.6) + github-pages-health-check (= 1.2.0) + jekyll (= 3.2.1) jekyll-coffeescript (= 1.0.1) jekyll-feed (= 0.5.1) jekyll-gist (= 1.4.0) - jekyll-github-metadata (= 2.0.2) - jekyll-mentions (= 1.1.3) + jekyll-github-metadata (= 2.1.0) + jekyll-mentions (= 1.2.0) jekyll-paginate (= 1.1.0) jekyll-redirect-from (= 0.11.0) jekyll-sass-converter (= 1.3.0) jekyll-seo-tag (= 2.0.0) jekyll-sitemap (= 0.10.0) + jekyll-swiss (= 0.4.0) jemoji (= 0.7.0) kramdown (= 1.11.1) liquid (= 3.0.6) listen (= 3.0.6) mercenary (~> 0.3) + minima (= 1.2.0) rouge (= 1.11.1) terminal-table (~> 1.4) - github-pages-health-check (1.1.0) + github-pages-health-check (1.2.0) addressable (~> 2.3) net-dns (~> 0.8) octokit (~> 4.0) @@ -51,13 +54,14 @@ GEM activesupport (>= 2) nokogiri (>= 1.4) i18n (0.7.0) - jekyll (3.1.6) - colorator (~> 0.1) + jekyll (3.2.1) + colorator (~> 1.0) jekyll-sass-converter (~> 1.0) jekyll-watch (~> 1.1) kramdown (~> 1.3) liquid (~> 3.0) mercenary (~> 0.3.3) + pathutil (~> 0.9) rouge (~> 1.7) safe_yaml (~> 1.0) jekyll-coffeescript (1.0.1) @@ -65,10 +69,11 @@ GEM jekyll-feed (0.5.1) jekyll-gist (1.4.0) octokit (~> 4.2) - jekyll-github-metadata (2.0.2) + jekyll-github-metadata (2.1.0) jekyll (~> 3.1) octokit (~> 4.0) - jekyll-mentions (1.1.3) + jekyll-mentions (1.2.0) + activesupport (~> 4.0) html-pipeline (~> 2.3) jekyll (~> 3.0) jekyll-paginate (1.1.0) @@ -79,6 +84,7 @@ GEM jekyll-seo-tag (2.0.0) jekyll (~> 3.1) jekyll-sitemap (0.10.0) + jekyll-swiss (0.4.0) jekyll-watch (1.5.0) listen (~> 3.0, < 3.1) jemoji (0.7.0) @@ -94,15 +100,16 @@ GEM rb-inotify (>= 0.9.7) mercenary (0.3.6) mini_portile2 (2.1.0) - minitest (5.9.0) + minima (1.2.0) + minitest (5.9.1) multipart-post (2.0.0) net-dns (0.8.0) - nokogiri (1.6.8) + nokogiri (1.6.8.1) mini_portile2 (~> 2.1.0) - pkg-config (~> 1.1.7) octokit (4.3.0) sawyer (~> 0.7.0, >= 0.5.3) - pkg-config (1.1.7) + pathutil (0.14.0) + forwardable-extended (~> 2.6) public_suffix (1.5.3) rb-fsevent (0.9.7) rb-inotify (0.9.7) @@ -113,15 +120,20 @@ GEM sawyer (0.7.0) addressable (>= 2.3.5, < 2.5) faraday (~> 0.8, < 0.10) - terminal-table (1.6.0) + terminal-table (1.7.3) + unicode-display_width (~> 1.1.1) thread_safe (0.3.5) typhoeus (0.8.0) ethon (>= 0.8.0) tzinfo (1.2.2) thread_safe (~> 0.1) + unicode-display_width (1.1.1) PLATFORMS ruby DEPENDENCIES - github-pages (= 89) + github-pages (= 97) + +BUNDLED WITH + 1.11.2 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 684a3fa..f6ede57 100644 --- a/README.md +++ b/README.md @@ -35,24 +35,16 @@ Below are the steps required to update the Fluo project website for a new releas ```bash cd fluo-website/ - mkdir -p docs/1.0.0-beta-1 + mkdir -p docs/fluo/1.0.0-beta-1 ./_scripts/convert-docs.py /path/to/fluo/docs/ /path/to/fluo-website/docs/fluo/1.0.0-beta-1/ ``` 2. Modify `docs/index.md` to point to new release and update the `latest_fluo_release` variable in `_config.yml`. -3. Run this command to generate Javadocs from your your Fluo release tag repo and - copy them to this repo. The command assumes that you have a tag in your Fluo - repo named after your release version (i.e 1.0.0-beta-1): - - ```bash - ./_scripts/gen-javadoc.sh 1.0.0-beta-1 /path/to/repo/fluo/modules/api /path/to/fluo-website/apidocs/fluo - ``` - -4. Modify `apidocs/index.md` to point to the new javadocs that you just generated. +3. Confirm that javadocs are hosted externally and modify `apidocs/index.md` to point to link to them. -5. Create a blog post announcing the release in `_posts/blog/` +4. Create release-notes, release, and blog post announcing the release in `_posts/` ## Apache Fluo Recipes releases @@ -68,15 +60,9 @@ Steps to update website for new Fluo Recipes release: 2. Modify `docs/index.md` to point to new release and update the `latest_recipes_release` variable in `_config.yml`. -3. Run this command to generate Javadocs from your Fluo release tag repo and - copy them to this repo. The command assumes that you have a tag in your Fluo - repo named after your release version (i.e 1.0.0-beta-1): - - ```bash - ./_scripts/gen-javadoc.sh 1.0.0-beta-1 /path/to/repo/fluo-recipes /path/to/fluo-website/apidocs/fluo-recipes - ``` +3. Confirm that javadocs are hosted externally and modify `apidocs/index.md` to point to link to them. -4. Modify `apidocs/index.md` to point to the new javadocs that you just generated. +4. Create release-notes, release, and blog post announcing the release in `_posts/` [Jekyll]: http://jekyllrb.com/ [Bundler]: http://bundler.io/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_config.yml ---------------------------------------------------------------------- diff --git a/_config.yml b/_config.yml index cf6346a..7540b0e 100644 --- a/_config.yml +++ b/_config.yml @@ -29,11 +29,11 @@ defaults: permalink: "/:categories/:title/" - scope: - path: "_posts/release-summaries" + path: "_posts/release" type: "posts" values: - layout: "release-summary" - category: "release-summaries" + layout: "release" + category: "release" permalink: "/:categories/:title/" - scope: @@ -54,6 +54,8 @@ latest_fluo_release: "1.0.0-incubating" latest_fluo_release_date: "October 4, 2016" latest_recipes_release: "1.0.0-beta-2" latest_recipes_release_date: "March 29, 2016" +fluo_api_base: "https://javadoc.io/doc/org.apache.fluo/fluo-api"; +fluo_api_static: "https://static.javadoc.io/org.apache.fluo/fluo-api"; whitelist: [jekyll-redirect-from] gems: [jekyll-redirect-from] http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_layouts/default.html ---------------------------------------------------------------------- diff --git a/_layouts/default.html b/_layouts/default.html index 57b383b..31a75e5 100644 --- a/_layouts/default.html +++ b/_layouts/default.html @@ -31,8 +31,10 @@ </div> <div class="collapse navbar-collapse js-navbar-collapse" style="margin-top: 20px"> <ul class="navbar-nav nav"> - <li><a href="{{ site.baseurl }}/docs/">Documentation</a></li> - <li><a href="{{ site.baseurl }}/download/">Download</a></li> + <li><a href="{{ site.baseurl }}/release/">Releases</a></li> + <li><a href="{{ site.baseurl }}/tour/">Tour</a></li> + <li><a href="{{ site.baseurl }}/docs/">Docs</a></li> + <li><a href="{{ site.baseurl }}/apidocs/">API</a></li> <li class="dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href="#">Community<span class="caret"></span></a> <ul class="dropdown-menu"> http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_layouts/release-summary.html ---------------------------------------------------------------------- diff --git a/_layouts/release-summary.html b/_layouts/release-summary.html deleted file mode 100644 index c59dcc5..0000000 --- a/_layouts/release-summary.html +++ /dev/null @@ -1,21 +0,0 @@ ---- -layout: default ---- -<div class="post-header"> - <h2>{{ page.title }}</h1> - <p class="text-muted">{{ page.date | date_to_string }}</p> -</div> - -{% if page.historical %} - <div class="alert alert-danger" role="alert">This release summary is for a release of Fluo that made before Apache incubation.</div> -{% endif %} - -<div class="post-content"> - {{ content }} -</div> - -<hr> - -<div> - <p>All Fluo release summaries can be found in the <a href="{{ site.baseurl }}/release-summaries/">archive</a></p> -</div> http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_layouts/release.html ---------------------------------------------------------------------- diff --git a/_layouts/release.html b/_layouts/release.html new file mode 100644 index 0000000..979ea0b --- /dev/null +++ b/_layouts/release.html @@ -0,0 +1,21 @@ +--- +layout: default +--- +<div class="post-header"> + <h2>{{ page.title }}</h1> + <p class="text-muted">{{ page.date | date_to_string }}</p> +</div> + +{% if page.historical %} + <div class="alert alert-danger" role="alert">This release was made before Apache incubation.</div> +{% endif %} + +<div class="post-content"> + {{ content }} +</div> + +<hr> + +<div> + <p>All releases can be found in the <a href="{{ site.baseurl }}/release/">release archive</a></p> +</div> http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/blog/2014-10-02-fluo-1.0.0-alpha-1-is-released.md ---------------------------------------------------------------------- diff --git a/_posts/blog/2014-10-02-fluo-1.0.0-alpha-1-is-released.md b/_posts/blog/2014-10-02-fluo-1.0.0-alpha-1-is-released.md index 93a5956..5b84539 100644 --- a/_posts/blog/2014-10-02-fluo-1.0.0-alpha-1-is-released.md +++ b/_posts/blog/2014-10-02-fluo-1.0.0-alpha-1-is-released.md @@ -3,7 +3,7 @@ title: "Fluo 1.0.0-alpha-1 is released!" date: 2014-10-02 15:30:00 redirect_from: /fluo-1.0.0-alpha-1-is-released/ --- -The Fluo project is happy to announce the [1.0.0-alpha-1][1] release of [Fluo][2]. +The Fluo project is happy to announce the [1.0.0-alpha-1][1] release of [Fluo][2]. This is the first release of Fluo and is not ready for production use. We invite developers to try it out, play with the [quickstart][3] & [phrasecount][4] examples, and contribute back in the form of bug reports, new feature requests, and pull requests. @@ -11,13 +11,13 @@ Fluo is an implementation of Google's [percolator paper][10], which adds large-s [Release notes][7] and [javadocs][11] are available. The [fluo jars][8] have been deployed to Maven Central -[1]: /release-summaries/1.0.0-alpha-1/ +[1]: /release/fluo-1.0.0-alpha-1/ [2]: https://github.com/fluo-io/fluo [3]: https://github.com/fluo-io/fluo-quickstart [4]: https://github.com/fluo-io/phrasecount [5]: http://hadoop.apache.org/docs/r2.5.1/hadoop-yarn/hadoop-yarn-site/YARN.html [6]: https://accumulo.apache.org/ -[7]: /release-notes/1.0.0-alpha-1/ +[7]: /release-notes/fluo-1.0.0-alpha-1/ [8]: http://search.maven.org/#search%7Cga%7C1%7Cfluo [9]: http://twill.incubator.apache.org/ [10]: http://research.google.com/pubs/pub36726.html http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/blog/2015-06-09-fluo-1.0.0-beta-1-is-released.md ---------------------------------------------------------------------- diff --git a/_posts/blog/2015-06-09-fluo-1.0.0-beta-1-is-released.md b/_posts/blog/2015-06-09-fluo-1.0.0-beta-1-is-released.md index 8c74610..488e909 100644 --- a/_posts/blog/2015-06-09-fluo-1.0.0-beta-1-is-released.md +++ b/_posts/blog/2015-06-09-fluo-1.0.0-beta-1-is-released.md @@ -16,14 +16,14 @@ data in [Accumulo]. [Release notes][notes] and [javadocs] are available. The Fluo [jars] have been deployed to Maven Central. -[1.0.0-beta-1]: /release-summaries/1.0.0-beta-1/ +[1.0.0-beta-1]: /release/fluo-1.0.0-beta-1/ [Fluo]: https://github.com/fluo-io/fluo [documentation]: /docs/fluo/1.0.0-beta-1/ [quickstart]: https://github.com/fluo-io/fluo-quickstart [phrasecount]: https://github.com/fluo-io/phrasecount [YARN]: http://hadoop.apache.org/docs/r2.5.1/hadoop-yarn/hadoop-yarn-site/YARN.html [Accumulo]: https://accumulo.apache.org/ -[notes]: /release-notes/1.0.0-beta-1/ +[notes]: /release-notes/fluo-1.0.0-beta-1/ [jars]: http://search.maven.org/#search%7Cga%7C1%7Cfluo [paper]: http://research.google.com/pubs/pub36726.html [javadocs]: /apidocs/fluo/1.0.0-beta-1/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/blog/2015-12-22-beta-2-pre-release-stress-test.md ---------------------------------------------------------------------- diff --git a/_posts/blog/2015-12-22-beta-2-pre-release-stress-test.md b/_posts/blog/2015-12-22-beta-2-pre-release-stress-test.md index f43ff80..2936d59 100644 --- a/_posts/blog/2015-12-22-beta-2-pre-release-stress-test.md +++ b/_posts/blog/2015-12-22-beta-2-pre-release-stress-test.md @@ -60,7 +60,7 @@ Further testing A long run of webindex will also be run on EC2 before releasing beta-2. [1]: /blog/2014/12/30/stress-test-long-run/ -[2]: /release-notes/1.0.0-beta-1/ +[2]: /release-notes/fluo-1.0.0-beta-1/ [3]: https://github.com/fluo-io/fluo-stress [4]: https://github.com/fluo-io/fluo/commit/c4789b3100092683b37c57c48ddd87993e84972c [5]: https://github.com/fluo-io/fluo-stress/commit/32edaf91138bb13b442632262c23e7f13f8fb17c http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/blog/2016-01-12-fluo-1.0.0-beta-2-is-released.md ---------------------------------------------------------------------- diff --git a/_posts/blog/2016-01-12-fluo-1.0.0-beta-2-is-released.md b/_posts/blog/2016-01-12-fluo-1.0.0-beta-2-is-released.md index 15c46d5..9350ae3 100644 --- a/_posts/blog/2016-01-12-fluo-1.0.0-beta-2-is-released.md +++ b/_posts/blog/2016-01-12-fluo-1.0.0-beta-2-is-released.md @@ -24,9 +24,9 @@ and [Stress][fluo-stress], are running well for long periods on Amazon EC2. We invite early adopters to try out the beta-2 release and help flush out problems before 1.0.0. -[1.0.0-beta-2]: /release-summaries/1.0.0-beta-2/ +[1.0.0-beta-2]: /release/fluo-1.0.0-beta-2/ [Fluo]: https://github.com/fluo-io/fluo -[notes]: /release-notes/1.0.0-beta-2/ +[notes]: /release-notes/fluo-1.0.0-beta-2/ [webindex]: https://github.com/fluo-io/webindex [fluo-recipes]: https://github.com/fluo-io/fluo-recipes [fluo-stress]: https://github.com/fluo-io/fluo-stress http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/blog/2016-09-27-1.0.0-incubating-is-released.md ---------------------------------------------------------------------- diff --git a/_posts/blog/2016-09-27-1.0.0-incubating-is-released.md b/_posts/blog/2016-09-27-1.0.0-incubating-is-released.md new file mode 100644 index 0000000..2addbe6 --- /dev/null +++ b/_posts/blog/2016-09-27-1.0.0-incubating-is-released.md @@ -0,0 +1,11 @@ +--- +title: "Fluo 1.0.0-incubating is released!" +date: 2016-09-27 15:30:00 +--- + +Apache Fluo is happy to announce a [1.0.0-incubating][rel] release. This is the first +release of Fluo as an Apache project and the fourth release for the project. Starting +with this release, Fluo will follow [semver][semver] for all future API changes. + +[semver]: http://semver.org/ +[rel]: /release/fluo-1.0.0-incubating/ http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2014-10-02-1.0.0-alpha-1.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2014-10-02-1.0.0-alpha-1.md b/_posts/release-notes/2014-10-02-1.0.0-alpha-1.md deleted file mode 100644 index e9b3e37..0000000 --- a/_posts/release-notes/2014-10-02-1.0.0-alpha-1.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: Fluo 1.0.0-alpha-1 Release Notes -date: 2014-10-02 15:30:00 -version: 1.0.0-alpha-1 -historical: true -redirect_from: /1.0.0-alpha-1-release-notes/ ---- - -Fluo 1.0.0-alpha-1 is the first official release of Fluo. It contains 83 initial tickets marked for this first milestone. The roadmap for this release included a working initial implementation, completing an initial pass at the client API (which will be finalized in 1.0.0), and writing a stress test application. This release has not been tested much at scale. More scale and peformance testing will be done for the beta release. - -## Significant features - -This alpha release contained a lot of initial features to help establish the basic implementation and client API. All of these features will be expected to become more robust over time. - -#### MiniFluo - -MiniAccumulo and MiniFluo make it easy to start a local Fluo instance for integration testing. MiniFluo also makes it easy for developers to experiment with Fluo locally without installing it on an Accumulo cluster. - -#### Garbage Collection - -Without garbage collection, the columnns in a Fluo table would grow without bound as updates are applied over time. Because of this, it's important to clean up old fully-processed (and no longer being read) transaction data. The initial implementation of garbage collection kept N versions of columns. [Fluo-8][8] added the ability to garbage collect all column versions that are no longer being read by transactions anywhere else in the cluster. - -#### MapReduce InputFormat - -The FluoInputFormat allows a specific snapshot to be read into a mapreduce job. It was established in [Fluo-7][7]. - -#### FluoFileOutputFormat and FluoOutputFormat - -The FluoFileOuputFormat enables the bulk ingest of a Fluo table using mapreduce by creating the Accumulo r-files in HDFS. The FluoOutputFormat pushes keys directly into the Accumulo tablet servers through the client API. [Fluo-35][35] added this feature. - -#### Fluo Workers and Oracle running in Yarn - -Fluo can use Yarn to manage fault tolerance and guarantee multiple worker processes and some number of Oracle processes are kept running across the cluster. [Apache Twill][twill] was used for the Yarn integration. [Fluo-11][11] added an application that will deploy Fluo to a Yarn cluster. - -#### Fluo API - -Fluo provides a fluent-builder API for creating transactions to read and mutate data of various different data types in the Fluo table. This API was finalized in [Fluo-23][23] and [Fluo-98][98] when it was moved into its own module. It includes a type layer (finished in [Fluo-153][153]) that makes it easier to work with different types, default values, null return types, and row/column maps. - -#### Stress test - -A [stress test][stress] was created that uses a tree to force collisions on transactions trying to mutate multiple columns at the same time. This was established in [Fluo-24][24]. - -Other important improvements and features: - -* [Fluo-1][1] - Create scripts for starting/stopping services -* [Fluo-3][3] - use Curator for leader election and leader discovery -* [Fluo-13][13] - Register clients that perform transactions in Zookeeper -* [Fluo-15][15] - Leader election for the Oracle Server -* [Fluo-16][16] - Allow observer notification to be configured in AbstractObserver class -* [Fluo-89][89] - OracleApp should allow configurable number of instances -* [Fluo-98][98] - Create a Range class for Fluo -* [Fluo-123][123] - Hide implementation of Bytes in the API -* [Fluo-130][130] - Expose MiniFluo in public API -* [Fluo-131][131] - Make LoaderExector implement closeable -* [Fluo-147][147] - Create fluo-accumulo module for Accumulo iterators and formatter -* [Fluo-175][175] - Add newTransaction() method to FluoClient -* [Fluo-186][186] - Add Exceptions to API - -## Testing - -All unit and integration tests passed. Two [stress tests][stress] were run successfully on a 20-node cluster with one test generating 2000 integers and another generating 5000 integers. The 5000 integer test took about 10 minutes to run after the MapReduce load of integers was complete. - -## Current Status of Percolator Implementation - -The following tables identify the features necessary for a full implementation of Percolator and the current state of Fluo with regards to the implementation. For alpha, a test to excercise some of these features at scale was created, however, it was not run at scale for long periods. One of the goals for beta is to have at least a 24hr run of the scale test, on at least 20 nodes, succeed. - -### Necessary features: - -| Percolator Feature | Details | Testing | -|----------|-------------------|------| -| ACID-compliant cross-row transactions | Provides distributed transactions with ACID guarantees that work across Accumulo rows. Rows are distributed across tablet servers. | IT,CST,[#240][240]<sup>1</sup> | -| Observers | Perform transactions by sending read/write RPCs to Accumulo tablet servers, which in turn send read/write RPCs to HDFS. | IT,CST | -| Worker Process | Scans Accumulo for changed columns and invokes necessary observers as function calls on the worker. | IT,CST | -| Timestamp Oracle | Provides strictly increasing timestamps; a property required for correct operation of the snapshot isolation protocol. | IT,CST | -| Oracle Failover | When an active Oracle process dies, timestamps should be served from a different Oracle. They should never go backwards. | IT,[#37][37],[#241][241]<sup>1</sup> | -| Lightweight Lock Service | Makes searches for dirty notifications more efficient. Zookeeper is used in this case. Should be low-latency, replicated, balanced, and persist to a backing store. | IT,CST | -| Multithreaded Transasctions | Allows highly parallel execution of transactions and synchronization only during get() and commit() upon which transactions actually collide. | IT,CST | -| Snapshot Isolation | Non-serializable isolation mostly implemented through MVCC provided through Accumulo's timestamp portion of the key. | IT,CST,[#240][240]<sup>1</sup> | -| Roll-back / Roll-forward semantics | Failures during the first phase of commit cause transactions to be rolled back while failures during the second phase can cause them to be rolled forward. | IT,[#240][240]<sup>1</sup> | - -<sup>1</sup> This feature has been at least partially tested. - -### Performance enhancements: - -| Enhancement | Details | Testing | -|----------|-------------------|------| -| Locality Group for NOTIFY column | Add an extra index in the underlying r-files for the notify column where it can be scanned faster by workers. | IT,CST | -| Worker scan conflict resolution | Workers register random tablets they choose to scan. When two workers generate the same tablet, one of them re-generates a new random tablet. | [#5][5] | -| Worker advertisement | Workers advertise through the lock service so that the system can know if they failed and any transactions upon which they are working can be rolled back or rolled forward | IT | -| Oracle batching | Timestamps get allocated in batch to workers. Each worker should share one connection to Oracle to limit RPCs across threads | IT,CST | -| Weak Notifications | Allows notification of a cell without the possibility of transactional conflict through a special type of column that can be notified but not written to | IT | -| Conditional Mutations | Allows READ-MODIFY-WRITE of a mutation in a single RPC. | IT,CST | -| Conditional Mutation batching | Delay sending of conditional mutations destined for same tablet server so they can be batched. Lowers number of RPCs, adds latency but provides better parallelism. | IT,CST | -| Read batching | Delaying of read operations on tablet server for batching | [#138][138] | -| Prefetching & read-ahead caching | Prediction made each time a column is read, which other columns will be read in the same transaction based on previous behavior. | [#12][12] | - -* IT = Integration tests, CST = Cluster stress tests -* Testing columns providing a ticket number have not yet been implemented. - -[ACCUMULO-1000]: https://issues.apache.org/jira/browse/ACCUMULO-1000 -[percolator]: http://research.google.com/pubs/pub36726.html -[twill]: http://twill.incubator.apache.org/ -[stress]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/stress/README.md -[241]: https://github.com/fluo-io/fluo/issues/241 -[240]: https://github.com/fluo-io/fluo/issues/240 -[186]: https://github.com/fluo-io/fluo/issues/186 -[175]: https://github.com/fluo-io/fluo/issues/175 -[153]: https://github.com/fluo-io/fluo/issues/153 -[147]: https://github.com/fluo-io/fluo/issues/147 -[138]: https://github.com/fluo-io/fluo/issues/138 -[131]: https://github.com/fluo-io/fluo/issues/131 -[130]: https://github.com/fluo-io/fluo/issues/130 -[123]: https://github.com/fluo-io/fluo/issues/123 -[98]: https://github.com/fluo-io/fluo/issues/98 -[89]: https://github.com/fluo-io/fluo/issues/89 -[37]: https://github.com/fluo-io/fluo/issues/37 -[35]: https://github.com/fluo-io/fluo/issues/35 -[24]: https://github.com/fluo-io/fluo/issues/24 -[23]: https://github.com/fluo-io/fluo/issues/23 -[16]: https://github.com/fluo-io/fluo/issues/16 -[15]: https://github.com/fluo-io/fluo/issues/15 -[13]: https://github.com/fluo-io/fluo/issues/13 -[12]: https://github.com/fluo-io/fluo/issues/12 -[11]: https://github.com/fluo-io/fluo/issues/11 -[8]: https://github.com/fluo-io/fluo/issues/8 -[7]: https://github.com/fluo-io/fluo/issues/7 -[5]: https://github.com/fluo-io/fluo/issues/5 -[3]: https://github.com/fluo-io/fluo/issues/3 -[1]: https://github.com/fluo-io/fluo/issues/1 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2014-10-02-fluo-1.0.0-alpha-1.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2014-10-02-fluo-1.0.0-alpha-1.md b/_posts/release-notes/2014-10-02-fluo-1.0.0-alpha-1.md new file mode 100644 index 0000000..02913da --- /dev/null +++ b/_posts/release-notes/2014-10-02-fluo-1.0.0-alpha-1.md @@ -0,0 +1,129 @@ +--- +title: Fluo 1.0.0-alpha-1 Release Notes +date: 2014-10-02 15:30:00 +version: fluo-1.0.0-alpha-1 +historical: true +redirect_from: /1.0.0-alpha-1-release-notes/ +--- + +Fluo 1.0.0-alpha-1 is the first official release of Fluo. It contains 83 initial tickets marked for this first milestone. The roadmap for this release included a working initial implementation, completing an initial pass at the client API (which will be finalized in 1.0.0), and writing a stress test application. This release has not been tested much at scale. More scale and peformance testing will be done for the beta release. + +## Significant features + +This alpha release contained a lot of initial features to help establish the basic implementation and client API. All of these features will be expected to become more robust over time. + +#### MiniFluo + +MiniAccumulo and MiniFluo make it easy to start a local Fluo instance for integration testing. MiniFluo also makes it easy for developers to experiment with Fluo locally without installing it on an Accumulo cluster. + +#### Garbage Collection + +Without garbage collection, the columnns in a Fluo table would grow without bound as updates are applied over time. Because of this, it's important to clean up old fully-processed (and no longer being read) transaction data. The initial implementation of garbage collection kept N versions of columns. [Fluo-8][8] added the ability to garbage collect all column versions that are no longer being read by transactions anywhere else in the cluster. + +#### MapReduce InputFormat + +The FluoInputFormat allows a specific snapshot to be read into a mapreduce job. It was established in [Fluo-7][7]. + +#### FluoFileOutputFormat and FluoOutputFormat + +The FluoFileOuputFormat enables the bulk ingest of a Fluo table using mapreduce by creating the Accumulo r-files in HDFS. The FluoOutputFormat pushes keys directly into the Accumulo tablet servers through the client API. [Fluo-35][35] added this feature. + +#### Fluo Workers and Oracle running in Yarn + +Fluo can use Yarn to manage fault tolerance and guarantee multiple worker processes and some number of Oracle processes are kept running across the cluster. [Apache Twill][twill] was used for the Yarn integration. [Fluo-11][11] added an application that will deploy Fluo to a Yarn cluster. + +#### Fluo API + +Fluo provides a fluent-builder API for creating transactions to read and mutate data of various different data types in the Fluo table. This API was finalized in [Fluo-23][23] and [Fluo-98][98] when it was moved into its own module. It includes a type layer (finished in [Fluo-153][153]) that makes it easier to work with different types, default values, null return types, and row/column maps. + +#### Stress test + +A [stress test][stress] was created that uses a tree to force collisions on transactions trying to mutate multiple columns at the same time. This was established in [Fluo-24][24]. + +Other important improvements and features: + +* [Fluo-1][1] - Create scripts for starting/stopping services +* [Fluo-3][3] - use Curator for leader election and leader discovery +* [Fluo-13][13] - Register clients that perform transactions in Zookeeper +* [Fluo-15][15] - Leader election for the Oracle Server +* [Fluo-16][16] - Allow observer notification to be configured in AbstractObserver class +* [Fluo-89][89] - OracleApp should allow configurable number of instances +* [Fluo-98][98] - Create a Range class for Fluo +* [Fluo-123][123] - Hide implementation of Bytes in the API +* [Fluo-130][130] - Expose MiniFluo in public API +* [Fluo-131][131] - Make LoaderExector implement closeable +* [Fluo-147][147] - Create fluo-accumulo module for Accumulo iterators and formatter +* [Fluo-175][175] - Add newTransaction() method to FluoClient +* [Fluo-186][186] - Add Exceptions to API + +## Testing + +All unit and integration tests passed. Two [stress tests][stress] were run successfully on a 20-node cluster with one test generating 2000 integers and another generating 5000 integers. The 5000 integer test took about 10 minutes to run after the MapReduce load of integers was complete. + +## Current Status of Percolator Implementation + +The following tables identify the features necessary for a full implementation of Percolator and the current state of Fluo with regards to the implementation. For alpha, a test to excercise some of these features at scale was created, however, it was not run at scale for long periods. One of the goals for beta is to have at least a 24hr run of the scale test, on at least 20 nodes, succeed. + +### Necessary features: + +| Percolator Feature | Details | Testing | +|----------|-------------------|------| +| ACID-compliant cross-row transactions | Provides distributed transactions with ACID guarantees that work across Accumulo rows. Rows are distributed across tablet servers. | IT,CST,[#240][240]<sup>1</sup> | +| Observers | Perform transactions by sending read/write RPCs to Accumulo tablet servers, which in turn send read/write RPCs to HDFS. | IT,CST | +| Worker Process | Scans Accumulo for changed columns and invokes necessary observers as function calls on the worker. | IT,CST | +| Timestamp Oracle | Provides strictly increasing timestamps; a property required for correct operation of the snapshot isolation protocol. | IT,CST | +| Oracle Failover | When an active Oracle process dies, timestamps should be served from a different Oracle. They should never go backwards. | IT,[#37][37],[#241][241]<sup>1</sup> | +| Lightweight Lock Service | Makes searches for dirty notifications more efficient. Zookeeper is used in this case. Should be low-latency, replicated, balanced, and persist to a backing store. | IT,CST | +| Multithreaded Transasctions | Allows highly parallel execution of transactions and synchronization only during get() and commit() upon which transactions actually collide. | IT,CST | +| Snapshot Isolation | Non-serializable isolation mostly implemented through MVCC provided through Accumulo's timestamp portion of the key. | IT,CST,[#240][240]<sup>1</sup> | +| Roll-back / Roll-forward semantics | Failures during the first phase of commit cause transactions to be rolled back while failures during the second phase can cause them to be rolled forward. | IT,[#240][240]<sup>1</sup> | + +<sup>1</sup> This feature has been at least partially tested. + +### Performance enhancements: + +| Enhancement | Details | Testing | +|----------|-------------------|------| +| Locality Group for NOTIFY column | Add an extra index in the underlying r-files for the notify column where it can be scanned faster by workers. | IT,CST | +| Worker scan conflict resolution | Workers register random tablets they choose to scan. When two workers generate the same tablet, one of them re-generates a new random tablet. | [#5][5] | +| Worker advertisement | Workers advertise through the lock service so that the system can know if they failed and any transactions upon which they are working can be rolled back or rolled forward | IT | +| Oracle batching | Timestamps get allocated in batch to workers. Each worker should share one connection to Oracle to limit RPCs across threads | IT,CST | +| Weak Notifications | Allows notification of a cell without the possibility of transactional conflict through a special type of column that can be notified but not written to | IT | +| Conditional Mutations | Allows READ-MODIFY-WRITE of a mutation in a single RPC. | IT,CST | +| Conditional Mutation batching | Delay sending of conditional mutations destined for same tablet server so they can be batched. Lowers number of RPCs, adds latency but provides better parallelism. | IT,CST | +| Read batching | Delaying of read operations on tablet server for batching | [#138][138] | +| Prefetching & read-ahead caching | Prediction made each time a column is read, which other columns will be read in the same transaction based on previous behavior. | [#12][12] | + +* IT = Integration tests, CST = Cluster stress tests +* Testing columns providing a ticket number have not yet been implemented. + +[ACCUMULO-1000]: https://issues.apache.org/jira/browse/ACCUMULO-1000 +[percolator]: http://research.google.com/pubs/pub36726.html +[twill]: http://twill.incubator.apache.org/ +[stress]: https://github.com/fluo-io/fluo/blob/1.0.0-alpha-1/modules/stress/README.md +[241]: https://github.com/fluo-io/fluo/issues/241 +[240]: https://github.com/fluo-io/fluo/issues/240 +[186]: https://github.com/fluo-io/fluo/issues/186 +[175]: https://github.com/fluo-io/fluo/issues/175 +[153]: https://github.com/fluo-io/fluo/issues/153 +[147]: https://github.com/fluo-io/fluo/issues/147 +[138]: https://github.com/fluo-io/fluo/issues/138 +[131]: https://github.com/fluo-io/fluo/issues/131 +[130]: https://github.com/fluo-io/fluo/issues/130 +[123]: https://github.com/fluo-io/fluo/issues/123 +[98]: https://github.com/fluo-io/fluo/issues/98 +[89]: https://github.com/fluo-io/fluo/issues/89 +[37]: https://github.com/fluo-io/fluo/issues/37 +[35]: https://github.com/fluo-io/fluo/issues/35 +[24]: https://github.com/fluo-io/fluo/issues/24 +[23]: https://github.com/fluo-io/fluo/issues/23 +[16]: https://github.com/fluo-io/fluo/issues/16 +[15]: https://github.com/fluo-io/fluo/issues/15 +[13]: https://github.com/fluo-io/fluo/issues/13 +[12]: https://github.com/fluo-io/fluo/issues/12 +[11]: https://github.com/fluo-io/fluo/issues/11 +[8]: https://github.com/fluo-io/fluo/issues/8 +[7]: https://github.com/fluo-io/fluo/issues/7 +[5]: https://github.com/fluo-io/fluo/issues/5 +[3]: https://github.com/fluo-io/fluo/issues/3 +[1]: https://github.com/fluo-io/fluo/issues/1 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2015-06-09-1.0.0-beta-1.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2015-06-09-1.0.0-beta-1.md b/_posts/release-notes/2015-06-09-1.0.0-beta-1.md deleted file mode 100644 index bac711a..0000000 --- a/_posts/release-notes/2015-06-09-1.0.0-beta-1.md +++ /dev/null @@ -1,177 +0,0 @@ ---- -title: Fluo 1.0.0-beta-1 Release Notes -date: 2015-06-09 18:30:00 -version: 1.0.0-beta-1 -historical: true -redirect_from: /1.0.0-beta-1-release-notes/ ---- - -Fluo 1.0.0-beta-1 is the second official release of Fluo. The release closed [133 tickets][tickets]. This release is not recommended for production use. - -## Significant features - -This release contains many new features that makes it easier to run, develop, and monitor Fluo applications. - -#### Simplified Fluo administration on a local machine or EC2 cluster - -Developers can now run Fluo and its dependencies on their local machine ([#92][92]) or an AWS EC2 cluster ([#356][356]) using a few simple commands. -This was done by creating two administration tools called [Fluo-dev][fluo-dev] and [Fluo-deploy][fluo-deploy] whose scripts and configuration reside in repos -separate from the Fluo code base. These tools allow developers to collaborate and share configuration for running Fluo. - -#### Transaction metrics are viewable using common monitoring tools - -Fluo now publishes metrics ([#20][20]) about transactions, collisions, and timestamps using [Dropwizard metrics][dropwizard]. These metrics are by default published -using JMX and are viewable using JConsole or JVisualVM. Fluo can also be configured to publish metrics to Graphite or Ganglia. View the [metrics documentation][metrics] -for more information. - -#### Improved processing of notifications in Fluo workers - -Fluo workers were refactored to separate the code used for finding and executing work. Each worker uses a single thread for finding -work ([#19][19]). A new method was introduced to partition work among workers using a hash+mod of notifications ([#282][282]). -The commit message for [4100e23][4100e23] contains a good description of some of the benefits and drawback of the current -hashing approach. - -#### Improved the deletion of observer notifications - -When a cell is deleted in Accumulo, a delete marker is inserted. Delete markers stay around until -all files in a tablet are compacted. For Fluo this could cause a lot of notification delete markers -to build up over time. To avoid this buildup, the way Fluo deletes notifications was changed in -([#457][457]). Accumulo delete markers are no longer used, Fluo now uses a custom delete marker for -notifications. The custom deleter marker allows Fluo to do analysis when Accumulo flushes memory to -disk and avoid writing many delete markers to persistent storage. - -#### Easier management of Fluo from the command line - -Fluo now provides different scripts (**fluo**, **mini-fluo**, & **local-fluo**) for managing Fluo using a YARN cluster, MiniFluo, or local processes. Several commands -were created for these scripts. A **scan** command allows users to print a snapshot of a Fluo table ([#319][319]). A **info** command shows locations of containers -when running Fluo in YARN ([#297][297]). A **classpath** command gives users a list of jars needed to execute Fluo client code ([#436][436]). A **wait** command will -sleep until all notifications are processed ([#434][434]). - -#### Support for running multiple Fluo applications on a single cluster - -Users can now run multiple Fluo applications using a single cluster ([#454][454]). This enables different Fluo users to share the same cluster. Fluo applications -can be started and stopped independently. Each application has its own configuration. - -#### Fluo build improvements - -On each build, all Java code is automatically formatted based on Google Java Style ([#479][479]). Also, checkstyle and findbugs will fail the build if certain -standards are not reached ([#185][185]). The POM is also sorted ([#493][493]). - -#### Organized Fluo code base - -The Fluo stress test was moved to its own [repo][fluo-stress] and is no longer a sub-module ([#385][385]). MiniFluo was moved from fluo-core to the fluo-mini -module/jar ([#439][439]). This reduced the number of dependencies in fluo-core. However, developers will now need to include the fluo-mini jar in their Maven -POM if they start MiniFluo. - -#### Fluo testing improvements - -Integration tests can now be run from Eclipse ([#322][322]). Several new [unit tests][unit] were created. - - -#### Other important improvements and bug fixes - -* [#470][470] - Replaced FluoFileOutputFormat with an Accumulo Key/Value generator -* [#460][460] - Reduced Fluo API module dependencies -* [#456][456] - Fixed bug with notifications being lost when processes died -* [#446][446] - Simplified log configuration and configure rolling log files in YARN -* [#442][442] - Reduced the number of curator clients in FluoAdmin -* [#383][383] - Improved transaction logging to help users debug collisions. See [debugging documentation][debug]. -* [#365][365] - Analyze Fluo code to see what non-public Accumulo APIs are used -* [#362][362] - Made API data objects immutable -* [#349][349] - Support application level configuration in fluo.properties -* [#342][342] - Add a configurable retry timeout to Fluo clients -* [#294][294] - Fluo now uses chroot suffix in its Zookeeper connection. -* [#293][293] - Add argument checking to FluoConfiguration -* [#244][244] - Make re-initialization easier for user - -## Testing - -A successful long stress test run was conducted using Fluo built from commit -[fb647dd][fb647dd]. The test ran very well and never fell behind like a -[previous long run][old-stress-run] of stress did. The test had the following -properties. - - * Initialized stress test using 1 billion random integers. - * Ran 150 incremental loads of 100 thousand integers. Slept 3 minutes between loads. - * Used 19 m3.xlarge nodes on EC2. 16 workers and 3 masters - * Configuration for the test committed and tagged in git : [fluo-deploy tag][fd-tag-1] and [fluo-stress tag][fs-tag-1] - * Opened two issues as a result of test [#499][499] and [fluo-stress#30][fs-30] - -Below is the trailing output from running the test. - - - *****Generating and loading incremental data set 148***** - *****Generating and loading incremental data set 149***** - *****Generating and loading incremental data set 150***** - *****Calculating # of unique integers using MapReduce***** - UNIQUE=1014486419 - *****Wait for Fluo to finish processing***** - 05:33:40.158 [main] INFO io.fluo.cluster.runner.AppRunner - The wait command will exit when all notifications are processed - 05:33:40.417 [Thread-3] INFO io.fluo.core.oracle.OracleClient - Connected to oracle at worker4:9913 - 05:33:41.308 [main] INFO io.fluo.cluster.runner.AppRunner - All processing has finished! - *****Printing # of unique integers calculated by Fluo***** - Total at root : 1014486419 - Nodes Scanned : 59605 - *****Verifying Fluo & MapReduce results match***** - Success! Fluo & MapReduce both calculated 1014486419 unique integers - -The test ran for a little over 12 hours. Below are two plots pulled from -graphite showing the number of notifications queued and transaction rate over -the entire test run. Load transactions are not included in the rate. The rate -is transactions per second. The accuracy of these plots is uncertain because -no graphite configuration changes were made. The plots do seem within the -ballpark. - -  - -[tickets]: https://github.com/fluo-io/fluo/issues?q=milestone%3A1.0.0-beta-1+is%3Aclosed -[fd-tag-1]: https://github.com/keith-turner/fluo-deploy/tree/beta-long-test-1 -[fs-tag-1]: https://github.com/keith-turner/fluo-stress/tree/beta-long-test-1 -[fluo-dev]: https://github.com/fluo-io/fluo-dev -[fluo-deploy]: https://github.com/fluo-io/fluo-deploy -[fluo-stress]: https://github.com/fluo-io/fluo-stress -[dropwizard]: https://dropwizard.github.io/metrics/3.1.0/ -[debug]: https://github.com/fluo-io/fluo/blob/1.0.0-beta-1/docs/applications.md#debugging-applications -[metrics]: https://github.com/fluo-io/fluo/blob/1.0.0-beta-1/docs/metrics.md -[old-stress-run]: /blog/2014/12/30/stress-test-long-run/ -[unit]: https://github.com/fluo-io/fluo/issues?utf8=%E2%9C%93&q=milestone%3A1.0.0-beta-1+is%3Aclosed+%22unit+test%22 -[4100e23]: https://github.com/fluo-io/fluo/commit/4100e236b8438350e30eda924a6360e2c722ae37 -[fb647dd]: https://github.com/fluo-io/fluo/commit/fb647dd6a470e8015654f8ed99b9196f5f49582a -[499]: https://github.com/fluo-io/fluo/issues/499 -[493]: https://github.com/fluo-io/fluo/issues/493 -[479]: https://github.com/fluo-io/fluo/issues/479 -[470]: https://github.com/fluo-io/fluo/issues/470 -[460]: https://github.com/fluo-io/fluo/issues/460 -[457]: https://github.com/fluo-io/fluo/issues/457 -[456]: https://github.com/fluo-io/fluo/issues/456 -[454]: https://github.com/fluo-io/fluo/issues/454 -[446]: https://github.com/fluo-io/fluo/issues/446 -[442]: https://github.com/fluo-io/fluo/issues/442 -[439]: https://github.com/fluo-io/fluo/issues/439 -[436]: https://github.com/fluo-io/fluo/issues/436 -[434]: https://github.com/fluo-io/fluo/issues/434 -[385]: https://github.com/fluo-io/fluo/issues/385 -[383]: https://github.com/fluo-io/fluo/issues/383 -[365]: https://github.com/fluo-io/fluo/issues/365 -[362]: https://github.com/fluo-io/fluo/issues/362 -[356]: https://github.com/fluo-io/fluo/issues/356 -[342]: https://github.com/fluo-io/fluo/issues/342 -[349]: https://github.com/fluo-io/fluo/issues/349 -[322]: https://github.com/fluo-io/fluo/issues/322 -[319]: https://github.com/fluo-io/fluo/issues/319 -[297]: https://github.com/fluo-io/fluo/issues/297 -[293]: https://github.com/fluo-io/fluo/issues/293 -[294]: https://github.com/fluo-io/fluo/issues/294 -[282]: https://github.com/fluo-io/fluo/issues/282 -[244]: https://github.com/fluo-io/fluo/issues/244 -[241]: https://github.com/fluo-io/fluo/issues/241 -[240]: https://github.com/fluo-io/fluo/issues/240 -[185]: https://github.com/fluo-io/fluo/issues/185 -[138]: https://github.com/fluo-io/fluo/issues/138 -[92]: https://github.com/fluo-io/fluo/issues/92 -[37]: https://github.com/fluo-io/fluo/issues/37 -[20]: https://github.com/fluo-io/fluo/issues/20 -[19]: https://github.com/fluo-io/fluo/issues/19 -[12]: https://github.com/fluo-io/fluo/issues/12 -[5]: https://github.com/fluo-io/fluo/issues/5 -[fs-30]: https://github.com/fluo-io/fluo-stress/issues/30 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2015-06-09-fluo-1.0.0-beta-1.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2015-06-09-fluo-1.0.0-beta-1.md b/_posts/release-notes/2015-06-09-fluo-1.0.0-beta-1.md new file mode 100644 index 0000000..108e312 --- /dev/null +++ b/_posts/release-notes/2015-06-09-fluo-1.0.0-beta-1.md @@ -0,0 +1,177 @@ +--- +title: Fluo 1.0.0-beta-1 Release Notes +date: 2015-06-09 18:30:00 +version: fluo-1.0.0-beta-1 +historical: true +redirect_from: /1.0.0-beta-1-release-notes/ +--- + +Fluo 1.0.0-beta-1 is the second official release of Fluo. The release closed [133 tickets][tickets]. This release is not recommended for production use. + +## Significant features + +This release contains many new features that makes it easier to run, develop, and monitor Fluo applications. + +#### Simplified Fluo administration on a local machine or EC2 cluster + +Developers can now run Fluo and its dependencies on their local machine ([#92][92]) or an AWS EC2 cluster ([#356][356]) using a few simple commands. +This was done by creating two administration tools called [Fluo-dev][fluo-dev] and [Fluo-deploy][fluo-deploy] whose scripts and configuration reside in repos +separate from the Fluo code base. These tools allow developers to collaborate and share configuration for running Fluo. + +#### Transaction metrics are viewable using common monitoring tools + +Fluo now publishes metrics ([#20][20]) about transactions, collisions, and timestamps using [Dropwizard metrics][dropwizard]. These metrics are by default published +using JMX and are viewable using JConsole or JVisualVM. Fluo can also be configured to publish metrics to Graphite or Ganglia. View the [metrics documentation][metrics] +for more information. + +#### Improved processing of notifications in Fluo workers + +Fluo workers were refactored to separate the code used for finding and executing work. Each worker uses a single thread for finding +work ([#19][19]). A new method was introduced to partition work among workers using a hash+mod of notifications ([#282][282]). +The commit message for [4100e23][4100e23] contains a good description of some of the benefits and drawback of the current +hashing approach. + +#### Improved the deletion of observer notifications + +When a cell is deleted in Accumulo, a delete marker is inserted. Delete markers stay around until +all files in a tablet are compacted. For Fluo this could cause a lot of notification delete markers +to build up over time. To avoid this buildup, the way Fluo deletes notifications was changed in +([#457][457]). Accumulo delete markers are no longer used, Fluo now uses a custom delete marker for +notifications. The custom deleter marker allows Fluo to do analysis when Accumulo flushes memory to +disk and avoid writing many delete markers to persistent storage. + +#### Easier management of Fluo from the command line + +Fluo now provides different scripts (**fluo**, **mini-fluo**, & **local-fluo**) for managing Fluo using a YARN cluster, MiniFluo, or local processes. Several commands +were created for these scripts. A **scan** command allows users to print a snapshot of a Fluo table ([#319][319]). A **info** command shows locations of containers +when running Fluo in YARN ([#297][297]). A **classpath** command gives users a list of jars needed to execute Fluo client code ([#436][436]). A **wait** command will +sleep until all notifications are processed ([#434][434]). + +#### Support for running multiple Fluo applications on a single cluster + +Users can now run multiple Fluo applications using a single cluster ([#454][454]). This enables different Fluo users to share the same cluster. Fluo applications +can be started and stopped independently. Each application has its own configuration. + +#### Fluo build improvements + +On each build, all Java code is automatically formatted based on Google Java Style ([#479][479]). Also, checkstyle and findbugs will fail the build if certain +standards are not reached ([#185][185]). The POM is also sorted ([#493][493]). + +#### Organized Fluo code base + +The Fluo stress test was moved to its own [repo][fluo-stress] and is no longer a sub-module ([#385][385]). MiniFluo was moved from fluo-core to the fluo-mini +module/jar ([#439][439]). This reduced the number of dependencies in fluo-core. However, developers will now need to include the fluo-mini jar in their Maven +POM if they start MiniFluo. + +#### Fluo testing improvements + +Integration tests can now be run from Eclipse ([#322][322]). Several new [unit tests][unit] were created. + + +#### Other important improvements and bug fixes + +* [#470][470] - Replaced FluoFileOutputFormat with an Accumulo Key/Value generator +* [#460][460] - Reduced Fluo API module dependencies +* [#456][456] - Fixed bug with notifications being lost when processes died +* [#446][446] - Simplified log configuration and configure rolling log files in YARN +* [#442][442] - Reduced the number of curator clients in FluoAdmin +* [#383][383] - Improved transaction logging to help users debug collisions. See [debugging documentation][debug]. +* [#365][365] - Analyze Fluo code to see what non-public Accumulo APIs are used +* [#362][362] - Made API data objects immutable +* [#349][349] - Support application level configuration in fluo.properties +* [#342][342] - Add a configurable retry timeout to Fluo clients +* [#294][294] - Fluo now uses chroot suffix in its Zookeeper connection. +* [#293][293] - Add argument checking to FluoConfiguration +* [#244][244] - Make re-initialization easier for user + +## Testing + +A successful long stress test run was conducted using Fluo built from commit +[fb647dd][fb647dd]. The test ran very well and never fell behind like a +[previous long run][old-stress-run] of stress did. The test had the following +properties. + + * Initialized stress test using 1 billion random integers. + * Ran 150 incremental loads of 100 thousand integers. Slept 3 minutes between loads. + * Used 19 m3.xlarge nodes on EC2. 16 workers and 3 masters + * Configuration for the test committed and tagged in git : [fluo-deploy tag][fd-tag-1] and [fluo-stress tag][fs-tag-1] + * Opened two issues as a result of test [#499][499] and [fluo-stress#30][fs-30] + +Below is the trailing output from running the test. + + + *****Generating and loading incremental data set 148***** + *****Generating and loading incremental data set 149***** + *****Generating and loading incremental data set 150***** + *****Calculating # of unique integers using MapReduce***** + UNIQUE=1014486419 + *****Wait for Fluo to finish processing***** + 05:33:40.158 [main] INFO io.fluo.cluster.runner.AppRunner - The wait command will exit when all notifications are processed + 05:33:40.417 [Thread-3] INFO io.fluo.core.oracle.OracleClient - Connected to oracle at worker4:9913 + 05:33:41.308 [main] INFO io.fluo.cluster.runner.AppRunner - All processing has finished! + *****Printing # of unique integers calculated by Fluo***** + Total at root : 1014486419 + Nodes Scanned : 59605 + *****Verifying Fluo & MapReduce results match***** + Success! Fluo & MapReduce both calculated 1014486419 unique integers + +The test ran for a little over 12 hours. Below are two plots pulled from +graphite showing the number of notifications queued and transaction rate over +the entire test run. Load transactions are not included in the rate. The rate +is transactions per second. The accuracy of these plots is uncertain because +no graphite configuration changes were made. The plots do seem within the +ballpark. + +  + +[tickets]: https://github.com/fluo-io/fluo/issues?q=milestone%3A1.0.0-beta-1+is%3Aclosed +[fd-tag-1]: https://github.com/keith-turner/fluo-deploy/tree/beta-long-test-1 +[fs-tag-1]: https://github.com/keith-turner/fluo-stress/tree/beta-long-test-1 +[fluo-dev]: https://github.com/fluo-io/fluo-dev +[fluo-deploy]: https://github.com/fluo-io/fluo-deploy +[fluo-stress]: https://github.com/fluo-io/fluo-stress +[dropwizard]: https://dropwizard.github.io/metrics/3.1.0/ +[debug]: https://github.com/fluo-io/fluo/blob/1.0.0-beta-1/docs/applications.md#debugging-applications +[metrics]: https://github.com/fluo-io/fluo/blob/1.0.0-beta-1/docs/metrics.md +[old-stress-run]: /blog/2014/12/30/stress-test-long-run/ +[unit]: https://github.com/fluo-io/fluo/issues?utf8=%E2%9C%93&q=milestone%3A1.0.0-beta-1+is%3Aclosed+%22unit+test%22 +[4100e23]: https://github.com/fluo-io/fluo/commit/4100e236b8438350e30eda924a6360e2c722ae37 +[fb647dd]: https://github.com/fluo-io/fluo/commit/fb647dd6a470e8015654f8ed99b9196f5f49582a +[499]: https://github.com/fluo-io/fluo/issues/499 +[493]: https://github.com/fluo-io/fluo/issues/493 +[479]: https://github.com/fluo-io/fluo/issues/479 +[470]: https://github.com/fluo-io/fluo/issues/470 +[460]: https://github.com/fluo-io/fluo/issues/460 +[457]: https://github.com/fluo-io/fluo/issues/457 +[456]: https://github.com/fluo-io/fluo/issues/456 +[454]: https://github.com/fluo-io/fluo/issues/454 +[446]: https://github.com/fluo-io/fluo/issues/446 +[442]: https://github.com/fluo-io/fluo/issues/442 +[439]: https://github.com/fluo-io/fluo/issues/439 +[436]: https://github.com/fluo-io/fluo/issues/436 +[434]: https://github.com/fluo-io/fluo/issues/434 +[385]: https://github.com/fluo-io/fluo/issues/385 +[383]: https://github.com/fluo-io/fluo/issues/383 +[365]: https://github.com/fluo-io/fluo/issues/365 +[362]: https://github.com/fluo-io/fluo/issues/362 +[356]: https://github.com/fluo-io/fluo/issues/356 +[342]: https://github.com/fluo-io/fluo/issues/342 +[349]: https://github.com/fluo-io/fluo/issues/349 +[322]: https://github.com/fluo-io/fluo/issues/322 +[319]: https://github.com/fluo-io/fluo/issues/319 +[297]: https://github.com/fluo-io/fluo/issues/297 +[293]: https://github.com/fluo-io/fluo/issues/293 +[294]: https://github.com/fluo-io/fluo/issues/294 +[282]: https://github.com/fluo-io/fluo/issues/282 +[244]: https://github.com/fluo-io/fluo/issues/244 +[241]: https://github.com/fluo-io/fluo/issues/241 +[240]: https://github.com/fluo-io/fluo/issues/240 +[185]: https://github.com/fluo-io/fluo/issues/185 +[138]: https://github.com/fluo-io/fluo/issues/138 +[92]: https://github.com/fluo-io/fluo/issues/92 +[37]: https://github.com/fluo-io/fluo/issues/37 +[20]: https://github.com/fluo-io/fluo/issues/20 +[19]: https://github.com/fluo-io/fluo/issues/19 +[12]: https://github.com/fluo-io/fluo/issues/12 +[5]: https://github.com/fluo-io/fluo/issues/5 +[fs-30]: https://github.com/fluo-io/fluo-stress/issues/30 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2016-01-12-1.0.0-beta-2.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2016-01-12-1.0.0-beta-2.md b/_posts/release-notes/2016-01-12-1.0.0-beta-2.md deleted file mode 100644 index c31eca2..0000000 --- a/_posts/release-notes/2016-01-12-1.0.0-beta-2.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -title: Fluo 1.0.0-beta-2 Release Notes -date: 2016-01-12 15:30:00 -version: 1.0.0-beta-2 -historical: true -redirect_from: /1.0.0-beta-2-release-notes/ ---- - -Fluo 1.0.0-beta-2 is the third official release of Fluo. This release closed -[48 tickets][tickets]. There is no upgrade path from 1.0.0-beta-1 to -1.0.0-beta-2. - -## Significant features - -This release contains many new features that makes it easier to run, develop, and monitor Fluo applications. - -#### Improved Fluo metrics that can be sent to InfluxDB and viewed in Grafana - -In [#569][569], [#570][570], & [#580][580], Fluo metrics and monitoring were refactored to fix several -bugs and allow metrics to be sent to [InfluxDB][influxdb] and viewed in [Grafana][grafana]. Fluo metrics -are still instrumented using [Dropwizard metrics][dropwizard] but in [#574][574] metrics configuration was -moved from its own file and to now reside in `fluo.properties`. While Fluo metrics can still be sent to -many different tools (i.e Graphite, Ganglia), Fluo now ships with configuration that can be used to configure -a Fluo dashboard in Grafana that queries InfluxDB. To set up the sending of Fluo metrics to InfluxDB/Grafana, -view this [documentation][grafana-docs] or consider using [Fluo-dev][fluo-dev] or [Zetten][zetten] -to run Fluo as they can install InfluxDB+Grafana and setup metrics for you. - -#### Improved Fluo administration - -Several commands were added to the `fluo` script which is used to administer Fluo. A `fluo exec` command -([#581][581]) was created to provide an easy way to execute application code using Fluo classes and dependencies. -A `fluo list` command ([#523][523]) was created to let users list all Fluo applications within a Fluo instance. -The `fluo scan` command now has a `--raw` option ([#597][597]) that prints Fluo data as stored in Accumulo. This -was accomplished by moving the Fluo formatter from Accumulo shell to the scan command. The scan command can now -format non-ascii characters as hex ([#568][568]). The `fluo new` command was improved to prevent users from -setting invalid Fluo application names ([#510][510]). A bug was fixed in the `fluo start` command that was causing -time outs when starting applications ([#562][562]). Finally, the `fluo` command can now be run without the `apps/` -directory configured for an application for most commands ([#524][524]). Only the `init` and `start` commands need -this directory configured. All other commands only require the default properties file to be configured at -`conf/fluo.properties`. - -#### Made Fluo work better with Spark - -Several changes were made to Fluo to allow it work better with Apache Spark. All Fluo data types now implement -`Serializable` and can be use in Spark RDDs ([#539][539]). Fluo data types also now implement `Comparable` which -allows RDDs of Fluo data types to be sorted ([#544][544]). Also, a no args constructor was created for the -MutableBytes data type to enable Kryo serialization in Spark ([#549][549]). Finally, a new InputFormat called -`FluoEntryInputFormat` was created that returns key/value entries and the existing `FluoInputFormat` that returns -rows was renamed `FluoRowInputFormat` ([#538][538],[#540][540]). - -#### Performance improvements - -A good bit of time was spent analyzing Fluo while it was running to determine -where time is spent when executing transactions. Based on this analysis, it -was found that a good bit of time was spent committing transactions. Changes -were made in Fluo and Accumulo in order to decrease commit time. For Fluo, the -following changes were made : - -* [#591][591] - Shared batch writer increases transaction history -* [#590][590] - Increased batch writer threads and made configurable -* [#589][589] - Added 2nd conditional writer and logging of commit times -* [#584][584] - Adjust number of conditional writer threads based on cluster size - -For Accumulo, changes are being made in [ACCUMULO-4066] to decrease the time it -takes to process conditional mutations. Conditional mutations are used when -Fluo commits a transaction. - -These changes resulted in nice improvements over beta-1 in testing. However there -is probably still room for improvement. More analysis is needed. - -#### API Changes - -Once Fluo 1.0.0 is released, all releases after that will follow -[semver](http://semver.org/). For now some small API changes are still being -made. The following API changes happened between beta-1 and beta-2. - -* [#566][566] - Added RowColumnValue and made Accumulo init code use it -* [#551][551] - Added method to get start timestamp of transaction -* [#550][550] - Changed setObservers() to addObservers() - -#### Other important improvements and bug fixes - -* [#598][598] - Upgraded Hadoop to 2.6.3 and Accumulo to 1.6.4 -* [#587][587] - Specified datasource for all graphs in fluo's Grafana dashboard -* [#586][586] - Added efficient and easy way to build Bytes objects -* [#578][578] - Plot nothing in Grafana when no data exists -* [#573][573] - Fixed issues building against Accumulo 1.8.0-SNAPSHOT -* [#561][561] - Stopped checkstyle mvn plugin from running at validate -* [#559][559] - Eventually drop deleted data -* [#558][558] - Added arguments to deploy command to skip findbugs, checkstyle, and auto-formatting -* [#556][556] - Make TravisCI deploy snapshot jars after successful builds -* [#552][552] - Made eclipse stop complaining about unknown plugins -* [#547][547] - Provide better documentation for LoaderExecutor -* [#535][535] - Upgraded Twill to 0.6.0-incubating -* [#520][520] - Consolidate all implementation properties into FluoConfigurationImpl -* [#518][518] - Make Oracle run on a random port -* [#513][513] - Unable to pass spaces to scan command -* [#495][495] - Add support for notifications to Fluo formatter - -## Testing - -For this release, a long run of the [Webindex][webindex] application was performed and is documented in -[a blog post][webindex-post]. A long run of [Fluo stress][fluo-stress] was run and documented in -[another blog post][stress-post]. - -[ACCUMULO-4066]: https://issues.apache.org/jira/browse/ACCUMULO-4066 -[tickets]: https://github.com/fluo-io/fluo/issues?q=milestone%3A1.0.0-beta-2+is%3Aclosed -[fluo-dev]: https://github.com/fluo-io/fluo-dev -[zetten]: https://github.com/fluo-io/zetten -[fluo-stress]: https://github.com/fluo-io/fluo-stress -[dropwizard]: https://dropwizard.github.io/metrics/3.1.0/ -[grafana-docs]: /docs/fluo/1.0.0-beta-2/grafana/ -[influxdb]: https://github.com/influxdata/influxdb -[grafana]: http://grafana.org/ -[webindex]: https://github.com/fluo-io/webindex -[stress-post]: /blog/2015/12/22/beta-2-pre-release-stress-test/ -[webindex-post]: /blog/2016/01/11/webindex-long-run/ -[598]: https://github.com/fluo-io/fluo/pull/598 -[597]: https://github.com/fluo-io/fluo/issues/597 -[591]: https://github.com/fluo-io/fluo/issues/591 -[590]: https://github.com/fluo-io/fluo/pull/590 -[589]: https://github.com/fluo-io/fluo/pull/589 -[587]: https://github.com/fluo-io/fluo/pull/587 -[586]: https://github.com/fluo-io/fluo/pull/586 -[584]: https://github.com/fluo-io/fluo/pull/584 -[581]: https://github.com/fluo-io/fluo/pull/581 -[580]: https://github.com/fluo-io/fluo/issues/580 -[578]: https://github.com/fluo-io/fluo/pull/578 -[574]: https://github.com/fluo-io/fluo/issues/574 -[573]: https://github.com/fluo-io/fluo/pull/573 -[570]: https://github.com/fluo-io/fluo/issues/570 -[569]: https://github.com/fluo-io/fluo/issues/569 -[568]: https://github.com/fluo-io/fluo/issues/568 -[566]: https://github.com/fluo-io/fluo/pull/566 -[562]: https://github.com/fluo-io/fluo/issues/562 -[561]: https://github.com/fluo-io/fluo/pull/561 -[559]: https://github.com/fluo-io/fluo/issues/559 -[558]: https://github.com/fluo-io/fluo/pull/558 -[556]: https://github.com/fluo-io/fluo/issues/556 -[552]: https://github.com/fluo-io/fluo/pull/552 -[551]: https://github.com/fluo-io/fluo/pull/551 -[550]: https://github.com/fluo-io/fluo/pull/550 -[549]: https://github.com/fluo-io/fluo/pull/549 -[547]: https://github.com/fluo-io/fluo/issues/547 -[544]: https://github.com/fluo-io/fluo/issues/544 -[540]: https://github.com/fluo-io/fluo/issues/540 -[539]: https://github.com/fluo-io/fluo/issues/539 -[538]: https://github.com/fluo-io/fluo/issues/538 -[535]: https://github.com/fluo-io/fluo/pull/535 -[524]: https://github.com/fluo-io/fluo/issues/524 -[523]: https://github.com/fluo-io/fluo/issues/523 -[520]: https://github.com/fluo-io/fluo/issues/520 -[518]: https://github.com/fluo-io/fluo/issues/518 -[513]: https://github.com/fluo-io/fluo/issues/513 -[510]: https://github.com/fluo-io/fluo/issues/510 -[495]: https://github.com/fluo-io/fluo/issues/495 -[20]: https://github.com/fluo-io/fluo/issues/20 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2016-01-12-fluo-1.0.0-beta-2.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2016-01-12-fluo-1.0.0-beta-2.md b/_posts/release-notes/2016-01-12-fluo-1.0.0-beta-2.md new file mode 100644 index 0000000..d694b1e --- /dev/null +++ b/_posts/release-notes/2016-01-12-fluo-1.0.0-beta-2.md @@ -0,0 +1,157 @@ +--- +title: Fluo 1.0.0-beta-2 Release Notes +date: 2016-01-12 15:30:00 +version: fluo-1.0.0-beta-2 +historical: true +redirect_from: /1.0.0-beta-2-release-notes/ +--- + +Fluo 1.0.0-beta-2 is the third official release of Fluo. This release closed +[48 tickets][tickets]. There is no upgrade path from 1.0.0-beta-1 to +1.0.0-beta-2. + +## Significant features + +This release contains many new features that makes it easier to run, develop, and monitor Fluo applications. + +#### Improved Fluo metrics that can be sent to InfluxDB and viewed in Grafana + +In [#569][569], [#570][570], & [#580][580], Fluo metrics and monitoring were refactored to fix several +bugs and allow metrics to be sent to [InfluxDB][influxdb] and viewed in [Grafana][grafana]. Fluo metrics +are still instrumented using [Dropwizard metrics][dropwizard] but in [#574][574] metrics configuration was +moved from its own file and to now reside in `fluo.properties`. While Fluo metrics can still be sent to +many different tools (i.e Graphite, Ganglia), Fluo now ships with configuration that can be used to configure +a Fluo dashboard in Grafana that queries InfluxDB. To set up the sending of Fluo metrics to InfluxDB/Grafana, +view this [documentation][grafana-docs] or consider using [Fluo-dev][fluo-dev] or [Zetten][zetten] +to run Fluo as they can install InfluxDB+Grafana and setup metrics for you. + +#### Improved Fluo administration + +Several commands were added to the `fluo` script which is used to administer Fluo. A `fluo exec` command +([#581][581]) was created to provide an easy way to execute application code using Fluo classes and dependencies. +A `fluo list` command ([#523][523]) was created to let users list all Fluo applications within a Fluo instance. +The `fluo scan` command now has a `--raw` option ([#597][597]) that prints Fluo data as stored in Accumulo. This +was accomplished by moving the Fluo formatter from Accumulo shell to the scan command. The scan command can now +format non-ascii characters as hex ([#568][568]). The `fluo new` command was improved to prevent users from +setting invalid Fluo application names ([#510][510]). A bug was fixed in the `fluo start` command that was causing +time outs when starting applications ([#562][562]). Finally, the `fluo` command can now be run without the `apps/` +directory configured for an application for most commands ([#524][524]). Only the `init` and `start` commands need +this directory configured. All other commands only require the default properties file to be configured at +`conf/fluo.properties`. + +#### Made Fluo work better with Spark + +Several changes were made to Fluo to allow it work better with Apache Spark. All Fluo data types now implement +`Serializable` and can be use in Spark RDDs ([#539][539]). Fluo data types also now implement `Comparable` which +allows RDDs of Fluo data types to be sorted ([#544][544]). Also, a no args constructor was created for the +MutableBytes data type to enable Kryo serialization in Spark ([#549][549]). Finally, a new InputFormat called +`FluoEntryInputFormat` was created that returns key/value entries and the existing `FluoInputFormat` that returns +rows was renamed `FluoRowInputFormat` ([#538][538],[#540][540]). + +#### Performance improvements + +A good bit of time was spent analyzing Fluo while it was running to determine +where time is spent when executing transactions. Based on this analysis, it +was found that a good bit of time was spent committing transactions. Changes +were made in Fluo and Accumulo in order to decrease commit time. For Fluo, the +following changes were made : + +* [#591][591] - Shared batch writer increases transaction history +* [#590][590] - Increased batch writer threads and made configurable +* [#589][589] - Added 2nd conditional writer and logging of commit times +* [#584][584] - Adjust number of conditional writer threads based on cluster size + +For Accumulo, changes are being made in [ACCUMULO-4066] to decrease the time it +takes to process conditional mutations. Conditional mutations are used when +Fluo commits a transaction. + +These changes resulted in nice improvements over beta-1 in testing. However there +is probably still room for improvement. More analysis is needed. + +#### API Changes + +Once Fluo 1.0.0 is released, all releases after that will follow +[semver](http://semver.org/). For now some small API changes are still being +made. The following API changes happened between beta-1 and beta-2. + +* [#566][566] - Added RowColumnValue and made Accumulo init code use it +* [#551][551] - Added method to get start timestamp of transaction +* [#550][550] - Changed setObservers() to addObservers() + +#### Other important improvements and bug fixes + +* [#598][598] - Upgraded Hadoop to 2.6.3 and Accumulo to 1.6.4 +* [#587][587] - Specified datasource for all graphs in fluo's Grafana dashboard +* [#586][586] - Added efficient and easy way to build Bytes objects +* [#578][578] - Plot nothing in Grafana when no data exists +* [#573][573] - Fixed issues building against Accumulo 1.8.0-SNAPSHOT +* [#561][561] - Stopped checkstyle mvn plugin from running at validate +* [#559][559] - Eventually drop deleted data +* [#558][558] - Added arguments to deploy command to skip findbugs, checkstyle, and auto-formatting +* [#556][556] - Make TravisCI deploy snapshot jars after successful builds +* [#552][552] - Made eclipse stop complaining about unknown plugins +* [#547][547] - Provide better documentation for LoaderExecutor +* [#535][535] - Upgraded Twill to 0.6.0-incubating +* [#520][520] - Consolidate all implementation properties into FluoConfigurationImpl +* [#518][518] - Make Oracle run on a random port +* [#513][513] - Unable to pass spaces to scan command +* [#495][495] - Add support for notifications to Fluo formatter + +## Testing + +For this release, a long run of the [Webindex][webindex] application was performed and is documented in +[a blog post][webindex-post]. A long run of [Fluo stress][fluo-stress] was run and documented in +[another blog post][stress-post]. + +[ACCUMULO-4066]: https://issues.apache.org/jira/browse/ACCUMULO-4066 +[tickets]: https://github.com/fluo-io/fluo/issues?q=milestone%3A1.0.0-beta-2+is%3Aclosed +[fluo-dev]: https://github.com/fluo-io/fluo-dev +[zetten]: https://github.com/fluo-io/zetten +[fluo-stress]: https://github.com/fluo-io/fluo-stress +[dropwizard]: https://dropwizard.github.io/metrics/3.1.0/ +[grafana-docs]: /docs/fluo/1.0.0-beta-2/grafana/ +[influxdb]: https://github.com/influxdata/influxdb +[grafana]: http://grafana.org/ +[webindex]: https://github.com/fluo-io/webindex +[stress-post]: /blog/2015/12/22/beta-2-pre-release-stress-test/ +[webindex-post]: /blog/2016/01/11/webindex-long-run/ +[598]: https://github.com/fluo-io/fluo/pull/598 +[597]: https://github.com/fluo-io/fluo/issues/597 +[591]: https://github.com/fluo-io/fluo/issues/591 +[590]: https://github.com/fluo-io/fluo/pull/590 +[589]: https://github.com/fluo-io/fluo/pull/589 +[587]: https://github.com/fluo-io/fluo/pull/587 +[586]: https://github.com/fluo-io/fluo/pull/586 +[584]: https://github.com/fluo-io/fluo/pull/584 +[581]: https://github.com/fluo-io/fluo/pull/581 +[580]: https://github.com/fluo-io/fluo/issues/580 +[578]: https://github.com/fluo-io/fluo/pull/578 +[574]: https://github.com/fluo-io/fluo/issues/574 +[573]: https://github.com/fluo-io/fluo/pull/573 +[570]: https://github.com/fluo-io/fluo/issues/570 +[569]: https://github.com/fluo-io/fluo/issues/569 +[568]: https://github.com/fluo-io/fluo/issues/568 +[566]: https://github.com/fluo-io/fluo/pull/566 +[562]: https://github.com/fluo-io/fluo/issues/562 +[561]: https://github.com/fluo-io/fluo/pull/561 +[559]: https://github.com/fluo-io/fluo/issues/559 +[558]: https://github.com/fluo-io/fluo/pull/558 +[556]: https://github.com/fluo-io/fluo/issues/556 +[552]: https://github.com/fluo-io/fluo/pull/552 +[551]: https://github.com/fluo-io/fluo/pull/551 +[550]: https://github.com/fluo-io/fluo/pull/550 +[549]: https://github.com/fluo-io/fluo/pull/549 +[547]: https://github.com/fluo-io/fluo/issues/547 +[544]: https://github.com/fluo-io/fluo/issues/544 +[540]: https://github.com/fluo-io/fluo/issues/540 +[539]: https://github.com/fluo-io/fluo/issues/539 +[538]: https://github.com/fluo-io/fluo/issues/538 +[535]: https://github.com/fluo-io/fluo/pull/535 +[524]: https://github.com/fluo-io/fluo/issues/524 +[523]: https://github.com/fluo-io/fluo/issues/523 +[520]: https://github.com/fluo-io/fluo/issues/520 +[518]: https://github.com/fluo-io/fluo/issues/518 +[513]: https://github.com/fluo-io/fluo/issues/513 +[510]: https://github.com/fluo-io/fluo/issues/510 +[495]: https://github.com/fluo-io/fluo/issues/495 +[20]: https://github.com/fluo-io/fluo/issues/20 http://git-wip-us.apache.org/repos/asf/incubator-fluo-website/blob/0495f99c/_posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md ---------------------------------------------------------------------- diff --git a/_posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md b/_posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md new file mode 100644 index 0000000..acf9596 --- /dev/null +++ b/_posts/release-notes/2016-09-27-fluo-1.0.0-incubating.md @@ -0,0 +1,117 @@ +--- +title: Fluo 1.0.0-incubating Release Notes +date: 2016-09-27 15:30:00 +version: fluo-1.0.0-incubating +--- + +Apache Fluo 1.0.0-incubating is the first release of Fluo as an Apache project and the fourth +release for the project. For information about changes before this release, see the +[release notes archive](/release-notes/). + +## Significant changes + +The 1.0.0-incubating release includes [167 commits][commits] that made since the last release +(1.0.0-beta-2). The significant changes are summarized below. + +#### Stabilized API + +Starting with this release, Apache Fluo's API will follow [semver]. The API is defined as +everything under the org.apache.fluo.api Java package. Given that the plan is to support this API +for a long time, a lot of API improvements were made since the beta-2 release. The following is a +list of significant API changes since beta-2. + + * [#772][772] : Get with default + * [#770][770] : Use varargs for Columns + * [#768][768] : Made Obsever and Application configuration consistent + * [#743][743] : Make data classes final + * [#723][723] : Removed FluoAdmin.InitOps() + * [#714][714] : Use Charsequence in API + * [#695][695] : Removed all 3rd party types from API + * [#680][680] : Changed package prefix from io.fluo to org.apache + * [#647][647] : Added Oracle and Worker to API + * [#639][639] : Improved scanning API + * [#626][626] : Added operation to get row+column pairs. Further improved by [#758][758]. + * [#119][119] - Consistently offer String in API + +#### Configurable classpath + +The previous releases of Fluo contained specific versions of Accumulo, Hadoop, Zookeeper, and other +Fluo dependencies. However, the actual dependencies needed should be determined by the existing +version of software where Fluo is run. Fluo changed to a model of making its classpath be +completely configurable by an administrator. Example classpath configuration and download scripts +ship with Fluo to help make setup easier. These script were only tested with specific versions of +Hadoop and Accumulo and may not work with all versions. These changes were made in [#706][706] and +[#687][687]. + +#### New Fluo version command + +Added a version command to the Fluo scripts in [#779][779]. This makes it easy to automatically +build shaded jars with the correct version of Fluo for use in Spark and Map Reduce. + +#### User and historical metrics + +Two major improvements related to metrics were made. Support for user metrics was added by +[#767][767]. This allows observers to easily report application specific metrics. In [#635][635] +the example InfluxDB and Grafana configurations were updated to show historical metrics. + +#### Improved transaction processing + +Transaction processing throughput was increased by changes in [#593][593] and some subsequent bug +fixes. These changes moved commit processing from the threads executing user code to an +asynchronous background process. This allows user threads to queue a transaction for commit an +immediately start working on another notification. This new process allows many more transactions +to be working through the commit process concurrently. Before the change a worker could only be +committing up to a few hundred transactions at any time. After the change, a single worker could +easily be committing tens of thousands of transactions at a time. These changes increase the latency of +individual transactions, which can have some drawbacks [#650][650]. For this release changes were +made in [#654][654] to use a priority queue for notifications in the worker. The changes in +[#654][654] partially solves the lock wait problem identified in [#650][650]. + +#### Improved Accumulo scanning + +Fluo utilizes server side Accumulo iterators. For this release these iterators were modified to +leverage seeking on the sever side. Before these changes the iterators used to sequentially scan +data. For popular cells that had a lot versions, this could be slow. These changes were made in +[#623][623]. + +## Other changes of interest + +* [#668][668] - Added method to append byte to BytesBuilder +* [#648][648] - Added documentation to help users if YARN is killing containers +* [#619][619] [#621][621] - Created Fluo logo + +## Testing + +A two day run of Webindex was started on September 7th using the latest snapshot version of Fluo. +This test used 11 EC2 m3.xlarge nodes. The test ran without problem. A good bit before that a [3 +day run](/blog/2016/05/17/webindex-long-run-2/) of Webindex was conducted. + +[semver]: http://semver.org/ +[commits]: https://github.com/apache/incubator-fluo/compare/1.0.0-beta-2...e1dbc60 +[779]: https://github.com/fluo-io/fluo/issues/779 +[772]: https://github.com/fluo-io/fluo/issues/772 +[770]: https://github.com/fluo-io/fluo/issues/770 +[768]: https://github.com/fluo-io/fluo/issues/768 +[767]: https://github.com/fluo-io/fluo/issues/767 +[758]: https://github.com/fluo-io/fluo/issues/758 +[743]: https://github.com/fluo-io/fluo/issues/743 +[723]: https://github.com/fluo-io/fluo/issues/723 +[714]: https://github.com/fluo-io/fluo/issues/714 +[706]: https://github.com/fluo-io/fluo/issues/706 +[695]: https://github.com/fluo-io/fluo/issues/695 +[687]: https://github.com/fluo-io/fluo/issues/687 +[680]: https://github.com/fluo-io/fluo/issues/680 +[668]: https://github.com/fluo-io/fluo/issues/668 +[654]: https://github.com/fluo-io/fluo/issues/654 +[650]: https://github.com/fluo-io/fluo/issues/650 +[648]: https://github.com/fluo-io/fluo/issues/648 +[647]: https://github.com/fluo-io/fluo/issues/647 +[639]: https://github.com/fluo-io/fluo/issues/639 +[635]: https://github.com/fluo-io/fluo/issues/635 +[626]: https://github.com/fluo-io/fluo/issues/626 +[623]: https://github.com/fluo-io/fluo/issues/623 +[621]: https://github.com/fluo-io/fluo/issues/621 +[619]: https://github.com/fluo-io/fluo/issues/619 +[593]: https://github.com/fluo-io/fluo/issues/593 +[119]: https://github.com/fluo-io/fluo/issues/119 +
