http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2014-04-29-logo-jlee3.md ---------------------------------------------------------------------- diff --git a/_posts/2014-04-29-logo-jlee3.md b/_posts/2014-04-29-logo-jlee3.md new file mode 100644 index 0000000..c034fca --- /dev/null +++ b/_posts/2014-04-29-logo-jlee3.md @@ -0,0 +1,10 @@ +--- +layout: post +title: Logo Entry No. 11 - Jennifer Lee +author: P. Taylor Goetz +--- + + + + +
http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2014-05-27-round1-results.md ---------------------------------------------------------------------- diff --git a/_posts/2014-05-27-round1-results.md b/_posts/2014-05-27-round1-results.md new file mode 100644 index 0000000..015e167 --- /dev/null +++ b/_posts/2014-05-27-round1-results.md @@ -0,0 +1,38 @@ +--- +layout: post +title: Logo Contest - Round 1 Results +author: P. Taylor Goetz +--- + +Round one of the Apache Storm logo contest is now complete and was a great success. We received votes from 7 PPMC members as well as 46 votes from the greater Storm community. + +We would like to extend a very special thanks to all those who took the time and effort to create and submit a logo proposal. We would also like to thank everyone who voted. + +## Results ## +The Storm PPMC and community votes were closely aligned, with the community vote resolving a PPMC tie for the 3rd finalist as shown in the result table below. + +The three finalists entering the final round are: + + * [No. 6 - Alec Bartos](/2014/04/23/logo-abartos.html) + * [No. 9 - Jennifer Lee](/2014/04/29/logo-jlee1.html) + * [No. 10 - Jennifer Lee](/2014/04/29/logo-jlee2.html) + +Congratulations Alec and Jennifer! + +Stay tuned for the final vote, which will be announced shortly. + +------ + +| Entry | PPMC | Community | +|:-----------------------------|-----:|----------:| +|1 - Patricia Forrest | 2 | 23 | +|2 - Samuel Quiñones | 3 | 6 | +|3- Shaan Shiv Suleman | 0 | 7 | +|4 - Richard Brownlie-Marshall | 0 | 6 | +|5 - Ziba Sayari | 0 | 9 | +|6 - Alec Bartos | 3 | 32 | +|7 - Calum Boustead | 0 | 0 | +|8 - Stefano Asili | 0 | 2 | +|9 - Jennifer Lee | 9 | 63 | +|10 - Jennifer Lee | 18 | 64 | +|11 - Jennifer Lee | 0 | 18 | \ No newline at end of file http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2014-06-17-contest-results.md ---------------------------------------------------------------------- diff --git a/_posts/2014-06-17-contest-results.md b/_posts/2014-06-17-contest-results.md new file mode 100644 index 0000000..4b6b7de --- /dev/null +++ b/_posts/2014-06-17-contest-results.md @@ -0,0 +1,24 @@ +--- +layout: post +title: Storm Logo Contest Results +author: P. Taylor Goetz +--- + +The Apache Storm logo contest is now complete and was a great success. We received votes from 7 PPMC members as well as 55 votes from the greater Storm community. Thank you to everyone who participated by voting. + +## The Winner ## +Congratulations to Jennifer Lee, whose [3rd entry](/2014/04/29/logo-jlee2.html) received the most points from both the PPMC as well as the Storm community. Final vote tallies are listed below. + + + +The Apache Storm project management team will now begin the process of finalizing the logo and making it official. + +Congratulations Jennifer, and thanks again to everyone who participated! + +------ + +| Entry | PPMC | Community | +|:-----------------------------|-----:|----------:| +|[6 - Alec Bartos](/2014/04/23/logo-abartos.html) | 2 | 41 | +|[9 - Jennifer Lee](/2014/04/29/logo-jlee1.html) | 7 | 111 | +|[10 - Jennifer Lee](/2014/04/29/logo-jlee2.html) | 26 | 123 | http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2014-06-25-storm092-released.md ---------------------------------------------------------------------- diff --git a/_posts/2014-06-25-storm092-released.md b/_posts/2014-06-25-storm092-released.md new file mode 100644 index 0000000..7f8c438 --- /dev/null +++ b/_posts/2014-06-25-storm092-released.md @@ -0,0 +1,137 @@ +--- +layout: post +title: Storm 0.9.2 released +author: P. Taylor Goetz +--- + +We are pleased to announce that Storm 0.9.2-incubating has been released and is available from [the downloads page](/downloads.html). This release includes many important fixes and improvements. + +Netty Transport Improvements +---------------------------- +Storm's Netty-based transport has been overhauled to significantly improve performance through better utilization of thread, CPU, and network resources, particularly in cases where message sizes are small. Storm contributor Sean Zhong ([@clockfly](https://github.com/clockfly)) deserves a great deal of credit not only for discovering, analyzing, documenting and fixing the root cause, but also for persevering through an extended review process and promptly addressing all concerns. + +Those interested in the technical details and evolution of this patch can find out more in the [JIRA ticket for STORM-297](https://issues.apache.org/jira/browse/STORM-297). + +Sean also discovered and fixed an [elusive bug](https://issues.apache.org/jira/browse/STORM-342) in Storm's usage of the Disruptor queue that could lead to out-of-order or lost messages. + +Many thanks to Sean for contributing these important fixes. + +Storm UI Improvements +--------------------- +This release also includes a number of improvements to the Storm UI service. Contributor Sriharsha Chintalapani([@harshach](https://github.com/harshach)) added a REST API to the Storm UI service to expose metrics and operations in JSON format, and updated the UI to use that API. + +The new REST API will make it considerably easier for other services to consume availabe cluster and topology metrics for monitoring and visualization applications. Kyle Nusbaum ([@knusbaum](https://github.com/knusbaum)) has already leveraged the REST API to create a topology visualization tool now included in Storm UI and illustrated in the screenshot below. + + + + + + + +In the visualization, spout components are represented as blue, while bolts are colored between green and red depending on their associated capacity metric. The width of the lines between the components represent the flow of tuples relative to the other visible streams. + +Kafka Spout +----------- +This is the first Storm release to include official support for consuming data from Kafka 0.8.x. In the past, development of Kafka spouts for Storm had become somewhat fragmented and finding an implementation that worked with certain versions of Storm and Kafka proved burdensome for some developers. This is no longer the case, as the `storm-kafka` module is now part of the Storm project and associated artifacts are released to official channels (Maven Central) along with Storm's other components. + +Thanks are due to GitHub user [@wurstmeister]() for picking up Nathan Marz' original Kafka 0.7.x implementation, updating it to work with Kafka 0.8.x, and contributing that work back to the Storm community. + +The `storm-kafka` module can be found in the `/external/` directory of the source tree and binary distributions. The `external` area has been set up to contain projects that while not required by Storm, are often used in conjunction with Storm to integrate with some other technology. Such projects also come with a maintenance committment from at least one Storm committer to ensure compatibility with Storm's main codebase as it evolves. + +The `storm-kafka` dependency is available now from Maven Central at the following coordinates: + + + groupId: org.apache.storm + artifactId: storm-kafka + version: 0.9.2-incubating + +Users, and Scala developers in particular, should note that the Kafka dependency is listed as `provided`. This allows users to choose a specific Scala version as described in the [project README](https://github.com/apache/incubator-storm/tree/v0.9.2-incubating/external/storm-kafka). + + +Storm Starter and Examples +-------------------------- + +Similar to the `external` section of the codebase, we have also added an `examples` directory and pulled in the `storm-starter` project to ensure it will be maintained in lock-step with Storm's main codebase. + +Thank you to Storm committer Michael G. Noll for his continued work in maintaining and improving the `storm-starter` project. + + +Plugable Serialization for Multilang +------------------------------------ +In previous versions of Storm, serialization of data to and from multilang components was limited to JSON, imposing somewhat of performance penalty. Thanks to a contribution from John Gilmore ([@jsgilmore](https://github.com/jsgilmore)) the serialization mechanism is now plugable and enables the use of more performant serialization frameworks like protocol buffers in addition to JSON. + + +Thanks +------ +Special thanks are due to all those who have contributed to Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated. + + +Changelog +--------- + + * STORM-352: [storm-kafka] PartitionManager does not save offsets to ZooKeeper + * STORM-66: send taskid on initial handshake + * STORM-342: Contention in Disruptor Queue which may cause out of order or lost messages + * STORM-338: Move towards idiomatic Clojure style + * STORM-335: add drpc test for removing timed out requests from queue + * STORM-69: Storm UI Visualizations for Topologies + * STORM-297: Performance scaling with CPU + * STORM-244: DRPC timeout can return null instead of throwing an exception + * STORM-63: remove timeout drpc request from its function's request queue + * STORM-313: Remove log-level-page from logviewer + * STORM-205: Add REST API To Storm UI + * STORM-326: tasks send duplicate metrics + * STORM-331: Update the Kafka dependency of storm-kafka to 0.8.1.1 + * STORM-308: Add support for config_value to {supervisor,nimbus,ui,drpc,logviewer} childopts + * STORM-309: storm-starter Readme: windows documentation update + * STORM-318: update storm-kafka to use apache curator-2.4.0 + * STORM-303: storm-kafka reliability improvements + * STORM-233: Removed inline heartbeat to nimbus to avoid workers being killed when under heavy ZK load + * STORM-267: fix package name of LoggingMetricsConsumer in storm.yaml.example + * STORM-265: upgrade to clojure 1.5.1 + * STORM-232: ship JNI dependencies with the topology jar + * STORM-295: Add storm configuration to define JAVA_HOME + * STORM-138: Pluggable serialization for multilang + * STORM-264: Removes references to the deprecated topology.optimize + * STORM-245: implement Stream.localOrShuffle() for trident + * STORM-317: Add SECURITY.md to release binaries + * STORM-310: Change Twitter authentication + * STORM-305: Create developer documentation + * STORM-280: storm unit tests are failing on windows + * STORM-298: Logback file does not include full path for metrics appender fileNamePattern + * STORM-316: added validation to registermetrics to have timebucketSizeInSecs >= 1 + * STORM-315: Added progress bar when submitting topology + * STORM-214: Windows: storm.cmd does not properly handle multiple -c arguments + * STORM-306: Add security documentation + * STORM-302: Fix Indentation for pom.xml in storm-dist + * STORM-235: Registering a null metric should blow up early + * STORM-113: making thrift usage thread safe for local cluster + * STORM-223: use safe parsing for reading YAML + * STORM-238: LICENSE and NOTICE files are duplicated in storm-core jar + * STORM-276: Add support for logviewer in storm.cmd. + * STORM-286: Use URLEncoder#encode with the encoding specified. + * STORM-296: Storm kafka unit tests are failing on windows + * STORM-291: upgrade http-client to 4.3.3 + * STORM-252: Upgrade curator to latest version + * STORM-294: Commas not escaped in command line + * STORM-287: Fix the positioning of documentation strings in clojure code + * STORM-290: Fix a log binding conflict caused by curator dependencies + * STORM-289: Fix Trident DRPC memory leak + * STORM-173: Treat command line "-c" option number config values as such + * STORM-194: Support list of strings in *.worker.childopts, handle spaces + * STORM-288: Fixes version spelling in pom.xml + * STORM-208: Add storm-kafka as an external module + * STORM-285: Fix storm-core shade plugin config + * STORM-12: reduce thread usage of netty transport + * STORM-281: fix and issue with config parsing that could lead to leaking file descriptors + * STORM-196: When JVM_OPTS are set, storm jar fails to detect storm.jar from environment + * STORM-260: Fix a potential race condition with simulated time in Storm's unit tests + * STORM-258: Update commons-io version to 2.4 + * STORM-270: don't package .clj files in release jars. + * STORM-273: Error while running storm topologies on Windows using "storm jar" + * STROM-247: Replace links to github resources in storm script + * STORM-263: Update Kryo version to 2.21+ + * STORM-187: Fix Netty error "java.lang.IllegalArgumentException: timeout value is negative" + * STORM-186: fix float secs to millis long convertion + * STORM-70: Upgrade to ZK-3.4.5 and curator-1.3.3 + * STORM-146: Unit test regression when storm is compiled with 3.4.5 zookeeper http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2014-10-20-storm093-release-candidate.md ---------------------------------------------------------------------- diff --git a/_posts/2014-10-20-storm093-release-candidate.md b/_posts/2014-10-20-storm093-release-candidate.md new file mode 100644 index 0000000..6270283 --- /dev/null +++ b/_posts/2014-10-20-storm093-release-candidate.md @@ -0,0 +1,11 @@ +--- +layout: post +title: Storm 0.9.3 release candidate 1 available +author: Michael G. Noll +--- + +We are pleased to announce that Apache Storm version 0.9.3 has reached the release candidate stage. The latest release is 0.9.3-rc1. We are moving closer to the GA release, so this is a good time to give this new 0.9.3 version a try! + +Storm 0.9.3 is a maintenance release with bug fixes and better stability as summarized in the [0.9.3 changelog](https://github.com/apache/storm/blob/master/CHANGELOG.md). On top of that version 0.9.3 continues to improve the integration with [Apache Kafka](http://kafka.apache.org/) as well as the [documentation of the core spout and Trident spout for Kafka](https://github.com/apache/storm/blob/master/external/storm-kafka/README.md), which will provide our users with the best Kafka connectivity for Storm to date. + +We heartily encourage you to [test the 0.9.3 release candidate](http://storm.apache.org/downloads.html) and provide your feedback regarding any issues via [our mailing lists](http://storm.apache.org/community.html), which is an easy and valuable way to contribute back to the Storm community and to help us moving to an official 0.9.3 release. You can find the [0.9.3 release candidate in our Downloads section](http://storm.apache.org/downloads.html). http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2014-11-25-storm093-released.md ---------------------------------------------------------------------- diff --git a/_posts/2014-11-25-storm093-released.md b/_posts/2014-11-25-storm093-released.md new file mode 100644 index 0000000..df3da75 --- /dev/null +++ b/_posts/2014-11-25-storm093-released.md @@ -0,0 +1,164 @@ +--- +layout: post +title: Storm 0.9.3 released +author: P. Taylor Goetz +--- + +We are pleased to announce that Apache Storm 0.9.3 has been released and is available from [the downloads page](/downloads.html). This release includes 100 new fixes and improvements from 62 individual contributors. + +Improved Kafka Integration +---------------------------- + +Apache Storm has supported [Apache Kafka](http://kafka.apache.org/) as a streaming data source since version 0.9.2-incubating. Version 0.9.3 brings a number of improvements to the Kafka integration including the ability to write data to one or more Kafka clusters and topics. + +The ability to both read from and write to Kafka further unlocks potential of the already powerful combination of Storm and Kafka. Storm users can now use Kafka as a source of and destination for streaming data. This allows for inter-topology communication, topology chaining, combining spout/bolt-based topologies with Trident-based data flows, and integration with any external system that supports data ingest from Kafka. + +More information about Storm's Kafka integration can be found in the [storm-kafka project documentation](https://github.com/apache/storm/blob/v0.9.3/external/storm-kafka/README.md). + + +HDFS Integration +---------------- + +Many stream processing use cases involve storing data in HDFS for further analysis and offline (batch) processing. Apache Stormâs HDFS integration consists of several bolt and Trident state implementations that allow topology developers to easily write data to HDFS from any Storm topology. + +More information about Storm's HDFS integration can be found in the [storm-hdfs project documentation](https://github.com/apache/storm/tree/v0.9.3/external/storm-hdfs). + + +HBase Integration +----------------- + +Apache Stormâs HBase integration includes a number of components that allow Storm topologies to both write to and query HBase in real-time. Many organizations use Apache HBase as part of their big data strategy for batch, interactive, and real-time workflows. Stormâs HBase integration allows users to leverage data assets in HBase as streaming queries, as well as using HBase as a destination for streaming computation results. + +More information about Storm's HBase integration can be found in the [storm-hbase project documentation](https://github.com/apache/storm/tree/v0.9.3/external/storm-hbase). + +Reduced Dependency Conflicts +---------------------------- +In previous Storm releases, it was not uncommon for users' topology dependencies to conflict with the libraries used by Storm. In Storm 0.9.3 several dependency packages that were common sources of conflicts have been package-relocated (shaded) to avoid this situation. Developers are free to use the Storm-packaged versions, or supply their own version. + +The following table lists the dependency package relocations: + +| Dependency | Original Package | Storm Package | +|:---|:---|:---| +| Apache Thrift | `org.apache.thrift` | `org.apache.thrift7` | +| Netty | `org.jboss.netty` | `org.apache.storm.netty` | +| Google Guava | `com.google.common` | `org.apache.storm.guava` | +| | `com.google.thirdparty` | `org.apache.storm.guava.thirdparty` | +| Apache HTTPClient | `org.apache.http` | `org.apache.storm.http` | +| Apache ZooKeeper | `org.apache.zookeeper` | `org.apache.storm.zookeeper` | +| Apache Curator | `org.apache.curator` | `org.apache.storm.curator` | + +Multi-Lang Improvements +----------------------- +Apache Storm 0.9.3 includes a new [Node.js](http://nodejs.org) multi-lang implementation that allows developers to write spouts and bolts in JavaScript. + +In addition to the Node.js implementation, the multi-lang protocol has been substantially improved in terms of robustness and error handling capabilities. As a result, **the multi-lang API has changed in a non-backward-compatible way**. Users with existing multi-lang topologies should consult the Python, Ruby, and JavaScript multi-lang examples to determine the impact prior to upgrading. + + +Thanks +------ +Special thanks are due to all those who have contributed to Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated. + + +Full Changelog +--------- + + * STORM-558: change "swap!" to "reset!" to fix assignment-versions in supervisor + * STORM-555: Storm json response should set charset to UTF-8 + * STORM-513: check heartbeat from multilang subprocess + * STORM-549: "topology.enable.message.timeouts" does nothing + * STORM-546: Local hostname configuration ignored by executor + * STORM-492: Test timeout should be configurable + * STORM-540: Change default time format in logs to ISO8601 in order to include timezone + * STORM-511: Storm-Kafka spout keeps sending fetch requests with invalid offset + * STORM-538: Guava com.google.thirdparty.publicsuffix is not shaded + * STORM-497: don't modify the netty server taskToQueueId mapping while the someone could be reading it. + * STORM-537: A worker reconnects infinitely to another dead worker (Sergey Tryuber) + * STORM-519: add tuple as an input param to HBaseValueMapper + * STORM-488: Exit with 254 error code if storm CLI is run with unknown command + * STORM-506: Do not count bolt acks & fails in total stats + * STORM-490: fix build under Windows + * STORM-439: Replace purl.js qith jquery URL plugin + * STORM-499: Document and clean up shaded dependncy resolution with maven + * STORM-210: Add storm-hbase module + * STORM-507: Topology visualization should not block ui + * STORM-504: Class used by `repl` command is deprecated. + * STORM-330: Implement storm exponential backoff stategy for netty client and curator + * STORM-461: exit-process! does not always exit the process, but throws an exception + * STORM-341: fix assignment sorting + * STORM-476: external/storm-kafka: avoid NPE on null message payloads + * STORM-424: fix error message related to kafka offsets + * STORM-454: correct documentation in STORM-UI-REST-API.md + * STORM-474: Reformat UI HTML code + * STORM-447: shade/relocate packages of dependencies that are common causes of dependency conflicts + * STORM-279: cluster.xml doesn't take in STORM_LOG_DIR values. + * STORM-380: Kafka spout: throw RuntimeException if a leader cannot be found for a partition + * STORM-435: Improve storm-kafka documentation + * STORM-405: Add kafka trident state so messages can be sent to kafka topic + * STORM-211: Add module for HDFS integration + * STORM-337: Expose managed spout ids publicly + * STORM-320: Support STORM_CONF_DIR environment variable to support + * STORM-360: Add node details for Error Topology and Component pages + * STORM-54: Per-Topology Classpath and Environment for Workers + * STORM-355: excluding outdated netty transitively included via curator + * STORM-183: Replacing RunTime.halt() with RunTime.exit() + * STORM-213: Decouple In-Process ZooKeeper from LocalCluster. + * STORM-365: Add support for Python 3 to storm command. + * STORM-332: Enable Kryo serialization in storm-kafka + * STORM-370: Add check for empty table before sorting dom in UI + * STORM-359: add logviewer paging and download + * STORM-372: Typo in storm_env.ini + * STORM-266: Adding shell process pid and name in the log message + * STORM-367: Storm UI REST api documentation. + * STORM-200: Proposal for Multilang's Metrics feature + * STORM-351: multilang python process fall into endless loop + * STORM-375: Smarter downloading of assignments by supervisors and workers + * STORM-328: More restrictive Config checks, strict range check within Utils.getInt() + * STORM-381: Replace broken jquery.tablesorter.min.js to latest + * STORM-312: add storm monitor tools to monitor throughtput interactively + * STORM-354: Testing: allow users to pass TEST-TIMEOUT-MS as param for complete-topology + * STORM-254: one Spout/Bolt can register metric twice with same name in different timeBucket + * STORM-403: heartbeats-to-nimbus in supervisor-test failed due to uninten... + * STORM-402: FileNotFoundException when using storm with apache tika + * STORM-364: The exception time display as default timezone. + * STORM-420: Missing quotes in storm-starter python code + * STORM-399: Kafka Spout defaulting to latest offset when current offset is older then 100k + * STORM-421: Memoize local hostname lookup in executor + * STORM-414: support logging level to multilang protocol spout and bolt + * STORM-321: Added a tool to see the current status of STORM JIRA and github pulls. + * STORM-415: validate-launched-once in supervisor-test can not handle multiple topologies + * STORM-155: Storm rebalancing code causes multiple topologies assigned to a single port + * STORM-419: Updated test so sort ordering is very explicit. + * STORM-406: Fix for reconnect logic in netty client. + * STORM-366: Add color span to most recent error and fix ui templates. + * STORM-369: topology summary page displays wrong order. + * STORM-239: Allow supervisor to operate in paths with spaces in them + * STORM-87: fail fast on ShellBolt exception + * STORM-417: Storm UI lost table sort style when tablesorter was updated + * STORM-396: Replace NullPointerException with IllegalArgumentExeption + * STORM-451: Latest storm does not build due to a pom.xml error in storm-hdfs pom.xml + * STORM-453: migrated to curator 2.5.0 + * STORM-458: sample spout uses incorrect name when connecting bolt + * STORM-455: Report error-level messages from ShellBolt children + * STORM-443: multilang log's loglevel protocol can cause hang + * STORM-449: Updated ShellBolt to not exit when shutting down. + * STORM-464: Simulated time advanced after test cluster exits causes intermittent test failures + * STORM-463: added static version of metrics helpers for Config + * STORM-376: Add compression to serialization + * STORM-437: Enforce utf-8 when multilang reads from stdin + * STORM-361: Add JSON-P support to Storm UI API + * STORM-373: Provide Additional String substitutions for *.worker.childopts + * STORM-274: Add support for command remoteconfvalue in storm.cmd + * STORM-132: sort supervisor by free slot in desending order + * STORM-472: Improve error message for non-completeable testing spouts + * STORM-401: handle InterruptedIOException properly. + * STORM-461: exit-process! does not always exit the process, but throws an exception instead + * STORM-475: Storm UI pages do not use UTF-8 + * STORM-336: Logback version should be upgraded + * STORM-386: nodejs multilang protocol implementation and examples + * STORM-500: Add Spinner when UI is loading stats from nimbus + * STORM-501: Missing StormSubmitter API + * STORM-493: Workers inherit storm.conf.file/storm.options properties of their supervisor + * STORM-498: make ZK connection timeout configurable in Kafka spout + * STORM-428: extracted ITuple interface + * STORM-508: Update DEVELOPER.md now that Storm has graduated from Incubator + * STORM-514: Update storm-starter README now that Storm has graduated from Incubator http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2015-03-25-storm094-released.md ---------------------------------------------------------------------- diff --git a/_posts/2015-03-25-storm094-released.md b/_posts/2015-03-25-storm094-released.md new file mode 100644 index 0000000..0119e38 --- /dev/null +++ b/_posts/2015-03-25-storm094-released.md @@ -0,0 +1,24 @@ +--- +layout: post +title: Storm 0.9.4 released +author: P. Taylor Goetz +--- + +The Apache Storm community is pleased to announce that version 0.9.4 has been released and is available from [the downloads page](/downloads.html). + +This is a maintenance release that includes a number of important bug fixes that improve Storm's stability and fault tolerance. We encourage users of previous versions to upgrade to this latest release. + + +Thanks +------ +Special thanks are due to all those who have contributed to Apache Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated. + + +Full Changelog +--------- + + * STORM-559: ZkHosts in README should use 2181 as port. + * STORM-682: supervisor should handle worker state corruption gracefully. + * STORM-693: when kafka bolt fails to write tuple, it should report error instead of silently acking. + * STORM-329: fix cascading Storm failure by improving reconnection strategy and buffering messages + * STORM-130: Supervisor getting killed due to java.io.FileNotFoundException: File '../stormconf.ser' does not exist. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2015-06-04-storm095-released.md ---------------------------------------------------------------------- diff --git a/_posts/2015-06-04-storm095-released.md b/_posts/2015-06-04-storm095-released.md new file mode 100644 index 0000000..1d018c0 --- /dev/null +++ b/_posts/2015-06-04-storm095-released.md @@ -0,0 +1,23 @@ +--- +layout: post +title: Storm 0.9.5 released +author: P. Taylor Goetz +--- + +The Apache Storm community is pleased to announce that version 0.9.5 has been released and is available from [the downloads page](/downloads.html). + +This is a maintenance release that includes a number of important bug fixes that improve Storm's stability and fault tolerance. We encourage users of previous versions to upgrade to this latest release. + + +Thanks +------ +Special thanks are due to all those who have contributed to Apache Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated. + + +Full Changelog +--------- + + * STORM-790: Log "task is null" instead of letting worker die when task is null in transfer-fn + * STORM-796: Add support for "error" command in ShellSpout + * STORM-745: fix storm.cmd to evaluate 'shift' correctly with 'storm jar' + * STORM-130: Supervisor getting killed due to java.io.FileNotFoundException: File '../stormconf.ser' does not exist. http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_posts/2015-06-15-storm0100-beta-released.md ---------------------------------------------------------------------- diff --git a/_posts/2015-06-15-storm0100-beta-released.md b/_posts/2015-06-15-storm0100-beta-released.md new file mode 100644 index 0000000..87dda1b --- /dev/null +++ b/_posts/2015-06-15-storm0100-beta-released.md @@ -0,0 +1,294 @@ +--- +layout: post +title: Storm 0.10.0 Beta Released +author: P. Taylor Goetz +--- + +Fast on the heals of the 0.9.5 maintenance release, the Apache Storm community is pleased to announce that Apache Storm 0.10.0-beta has been released and is now available on [the downloads page](/downloads.html). + +Aside from many stability and performance improvements, this release includes a number of important new features, some of which are highlighted below. + +Secure, Multi-Tenant Deployment +-------------------------------- + +Much like the early days of Hadoop, Apache Storm originally evolved in an environment where security was not a high-priority concern. Rather, it was assumed that Storm would be deployed to environments suitably cordoned off from security threats. While a large number of users were comfortable setting up their own security measures for Storm (usually at the Firewall/OS level), this proved a hindrance to broader adoption among larger enterprises where security policies prohibited deployment without specific safeguards. + +Yahoo! hosts one of the largest Storm deployments in the world, and their engineering team recognized the need for security early on, so it implemented many of the features necessary to secure its own Apache Storm deployment. Yahoo!, Hortonworks, Symantec, and the broader Apache Storm community have worked together to bring those security innovations into the main Apache Storm code base. + +We are pleased to announce that work is now complete. Some of the highlights of Storm's new security features include: + + * Kerberos Authentication with Automatic Credential Push and Renewal + * Pluggable Authorization and ACLs + * Multi-Tenant Scheduling with Per-User isolation and configurable resource limits. + * User Impersonation + * SSL Support for Storm UI, Log Viewer, and DRPC (Distributed Remote Procedure Call) + * Secure integration with other Hadoop Projects (such as ZooKeeper, HDFS, HBase, etc.) + * User isolation (Storm topologies run as the user who submitted them) + +For more details and instructions for securing Storm, please see [the security documentation](https://github.com/apache/storm/blob/v0.10.0-beta/SECURITY.md). + +A Foundation for Rolling Upgrades and Continuity of Operations +-------------------------------------------------------------- + +In the past, upgrading a Storm cluster could be an arduous process that involved un-deploying existing topologies, removing state from local disk and ZooKeeper, installing the upgrade, and finally redeploying topologies. From an operations perspective, this process was disruptive to say the very least. + +The underlying cause of this headache was rooted in the data format Storm processes used to store both local and distributed state. Between versions, these data structures would change in incompatible ways. + +Beginning with version 0.10.0, this limitation has been eliminated. In the future, upgrading from Storm 0.10.0 to a newer version can be accomplished seamlessly, with zero down time. In fact, for users who use [Apache Ambari](https://ambari.apache.org) for cluster provisioning and management, the process can be completely automated. + + +Easier Deployment and Declarative Topology Wiring with Flux +---------------------------------------------------------- +Apache Storm 0.10.0 now includes Flux, which is a framework and set of utilities that make defining and deploying Storm topologies less painful and developer-intensive. A common pain point mentioned by Storm users is the fact that the wiring for a Topology graph is often tied up in Java code, and that any changes require recompilation and repackaging of the topology jar file. Flux aims to alleviate that pain by allowing you to package all your Storm components in a single jar, and use an external text file to define the layout and configuration of your topologies. + +Some of Flux' features include: + + * Easily configure and deploy Storm topologies (Both Storm core and Micro-batch API) without embedding configuration in your topology code + * Support for existing topology code + * Define Storm Core API (Spouts/Bolts) using a flexible YAML DSL + * YAML DSL support for most Storm components (storm-kafka, storm-hdfs, storm-hbase, etc.) + * Convenient support for multi-lang components + * External property substitution/filtering for easily switching between configurations/environments (similar to Maven-style `${variable.name}` substitution) + +You can read more about Flux on the [Flux documentation page](https://github.com/apache/storm/blob/v0.10.0-beta/external/flux/README.md). + +Partial Key Groupings +--------------------- +In addition to the standard Stream Groupings Storm has always supported, version 0.10.0 introduces a new grouping named "Partial Key Grouping". With the Partial Stream Grouping, the tuple stream is partitioned by the fields specified in the grouping, like the Fields Grouping, but are load balanced between two downstream bolts, which provides better utilization of resources when the incoming data is skewed. + +Documentation for the Partial Key Grouping and other stream groupings supported by Storm can be found [here](https://storm.apache.org/documentation/Concepts.html). [This research paper](https://melmeric.files.wordpress.com/2014/11/the-power-of-both-choices-practical-load-balancing-for-distributed-stream-processing-engines.pdf) provides additional details regarding how it works and its advantages. + + +Improved Logging Framework +-------------------------- +Debugging distributed applications can be difficult, and usually focuses on one main source of information: application log files. But in a very low latency system like Storm where every millisecond counts, logging can be a double-edged sword: If you log too little information you may miss the information you need to solve a problem; log too much and you risk degrading the overall performance of your application as resources are consumed by the logging framework. + +In version 0.10.0 Storm's logging framework now uses [Apache Log4j 2](http://logging.apache.org/log4j/2.x/) which, like Storm's internal messaging subsystem, uses the extremely performant [LMAX Disruptor](https://lmax-exchange.github.io/disruptor/) messaging library. Log4j 2 boast an 18x higher throughput and orders of magnitude lower latency than Storm's previous logging framework. More efficient resource utilization at the logging level means more resources are available where they matter most: executing your business logic. + +A few of the important features these changes bring include: + + * Rolling log files with size, duration, and date-based triggers that are composable + * Dynamic log configuration updates without dropping log messages + * Remote log monitoring and (re)configuration via JMX + * A Syslog/[RFC-5424](https://tools.ietf.org/html/rfc5424)-compliant appender. + * Integration with log aggregators such as syslog-ng + + +Streaming ingest with Apache Hive +--------------------------------- +Introduced in version 0.13, [Apache Hive](https://hive.apache.org) includes a [Streaming Data Ingest API](https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest) that allows data to be written continuously into Hive. The incoming data can be continuously committed in small batches of records into existing Hive partition or table. Once the data is committed its immediately visible to all hive queries. + +Apache Storm 0.10.0 introduces both a Storm Core API bolt implementation +that allows users to stream data from Storm directly into hive. Storm's Hive integration also includes a [State](https://storm.apache.org/documentation/Trident-state) implementation for Storm's Micro-batching/Transactional API (Trident) that allows you to write to Hive from a micro-batch/transactional topology and supports exactly-once semantics for data persistence. + +For more information on Storm's Hive integration, see the [storm-hive documentation](https://github.com/apache/storm/blob/v0.10.0-beta/external/storm-hive/README.md). + + +Microsoft Azure Event Hubs Integration +-------------------------------------- +With Microsoft Azure's [support for running Storm on HDInsight](https://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-overview/), Storm is now a first class citizen of the Azure cloud computing platform. To better support Storm integration with Azure services, Microsoft engineers have contributed several components that allow Storm to integrate directly with [Microsoft Azure Event Hubs](http://azure.microsoft.com/en-us/services/event-hubs/). + +Storm's Event Hubs integration includes both spout and bolt implementations for reading from, and writing to Event Hubs. The Event Hub integration also includes a Micro-batching/Transactional (Trident) spout implementation that supports fully fault-tolerant and reliable processing, as well as support for exactly-once message processing semantics. + + +Redis Support +------------- +Apache Storm 0.10.0 also introduces support for the [Redis](http://redis.io) data structure server. Storm's Redis support includes bolt implementations for both writing to and querying Redis from a Storm topology, and is easily extended for custom use cases. For Storm's micro-batching/transactional API, the Redis support includes both [Trident State and MapState](https://storm.apache.org/documentation/Trident-state.html) implementations for fault-tolerant state management with Redis. + +Further information can be found in the [storm-redis documentation](https://github.com/apache/storm/blob/v0.10.0-beta/external/storm-redis/README.md). + + +JDBC/RDBMS Integration +---------------------- +Many stream processing data flows require accessing data from or writing data to a relational data store. Storm 0.10.0 introduces highly flexible and customizable support for integrating with virtually any JDBC-compliant database. + +The Storm-JDBC package includes core Storm bolt and Trident state implementations that allow a storm topology to either insert Storm tuple data into a database table or execute select queries against a database to enrich streaming data in a storm topology. + +Further details and instructions can be found in the [Storm-JDBC documentation](https://github.com/apache/storm/blob/v0.10.0-beta/external/storm-jdbc/README.md). + + +Reduced Dependency Conflicts +---------------------------- +In previous Storm releases, it was not uncommon for users' topology dependencies to conflict with the libraries used by Storm. In Storm 0.9.3 several dependency packages that were common sources of conflicts have been package-relocated (shaded) to avoid this situation. In 0.10.0 this list has been expanded. + +Developers are free to use the Storm-packaged versions, or supply their own version. + +The full list of Storm's package relocations can be found [here](https://github.com/apache/storm/blob/v0.10.0-beta/storm-core/pom.xml#L439). + + +Future Work +----------- +While the 0.10.0 release is an important milestone in the evolution of Apache Storm, the Storm community is actively working on new improvements, both near and long term, and continuously exploring the realm of the possible. + +Twitter recently announced the Heron project, which claims to provide substantial performance improvements while maintaining 100% API compatibility with Storm. The corresponding research paper provides additional details regarding the architectural improvements. The fact that Twitter chose to maintain API compatibility with Storm is a testament to the power and flexibility of that API. Twitter has also expressed a desire to share their experiences and work with the Apache Storm community. + +A number of concepts expressed in the Heron paper were already in the implementation stage by the Storm community even before it was published, and we look forward to working with Twitter to bring those and other improvements to Storm. + +Thanks +------ +Special thanks are due to all those who have contributed to Apache Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are very much valued and appreciated. + + +Full Change Log +--------------- + + * STORM-856: use serialized value of delay secs for topo actions + * STORM-852: Replaced Apache Log4j Logger with SLF4J API + * STORM-813: Change storm-starter's README so that it explains mvn exec:java cannot run multilang topology + * STORM-853: Fix upload API to handle multi-args properly + * STORM-850: Convert storm-core's logback-test.xml to log4j2-test.xml + * STORM-848: Shade external dependencies + * STORM-849: Add storm-redis to storm binary distribution + * STORM-760: Use JSON for serialized conf + * STORM-833: Logging framework logback -> log4j 2.x + * STORM-842: Drop Support for Java 1.6 + * STORM-835: Netty Client hold batch object until io operation complete + * STORM-827: Allow AutoTGT to work with storm-hdfs too. + * STORM-821: Adding connection provider interface to decouple jdbc connector from a single connection pooling implementation. + * STORM-818: storm-eventhubs configuration improvement and refactoring + * STORM-816: maven-gpg-plugin does not work with gpg 2.1 + * STORM-811: remove old metastor_db before running tests again. + * STORM-808: allow null to be parsed as null + * STORM-807: quote args to storm.py correctly + * STORM-801: Add Travis CI badge to README + * STORM-797: DisruptorQueueTest has some race conditions in it. + * STORM-796: Add support for "error" command in ShellSpout + * STORM-795: Update the user document for the extlib issue + * STORM-792: Missing documentation in backtype.storm.generated.Nimbus + * STORM-791: Storm UI displays maps in the config incorrectly + * STORM-790: Log "task is null" instead of let worker died when task is null in transfer-fn + * STORM-789: Send more topology context to Multi-Lang components via initial handshake + * STORM-788: UI Fix key for process latencies + * STORM-787: test-ns should announce test failures with 'BUILD FAILURE' + * STORM-786: KafkaBolt should ack tick tuples + * STORM-773: backtype.storm.transactional-test fails periodically with timeout + * STORM-772: Tasts fail periodically with InterruptedException or InterruptedIOException + * STORM-766: Include version info in the service page + * STORM-765: Thrift serialization for local state + * STORM-764: Have option to compress thrift heartbeat + * STORM-762: uptime for worker heartbeats is lost when converted to thrift + * STORM-761: An option for new/updated Redis keys to expire in RedisMapState + * STORM-757: Simulated time can leak out on errors + * STORM-753: Improve RedisStateQuerier to convert List<Values> from Redis value + * STORM-752: [storm-redis] Clarify Redis*StateUpdater's expire is optional + * STORM-750: Set Config serialVersionUID + * STORM-749: Remove CSRF check from the REST API. + * STORM-747: assignment-version-callback/info-with-version-callback are not fired when assignments change + * STORM-746: Skip ack init when there are no output tasks + * STORM-745: fix storm.cmd to evaluate 'shift' correctly with 'storm jar' + * STORM-741: Allow users to pass a config value to perform impersonation. + * STORM-740: Simple Transport Client cannot configure thrift buffer size + * STORM-737: Check task->node+port with read lock to prevent sending to closed connection + * STORM-735: [storm-redis] Upgrade Jedis to 2.7.0 + * STORM-730: remove extra curly brace + * STORM-729: Include Executors (Window Hint) if the component is of Bolt type + * STORM-728: Put emitted and transferred stats under correct columns + * STORM-727: Storm tests should succeed even if a storm process is running locally. + * STORM-724: Document RedisStoreBolt and RedisLookupBolt which is missed. + * STORM-723: Remove RedisStateSetUpdater / RedisStateSetCountQuerier which didn't tested and have a bug + * STORM-721: Storm UI server should support SSL. + * STORM-715: Add a link to AssignableMetric.java in Metrics.md + * STORM-714: Make CSS more consistent with self, prev release + * STORM-713: Include topic information with Kafka metrics. + * STORM-712: Storm daemons shutdown if OutOfMemoryError occurs in any thread + * STORM-711: All connectors should use collector.reportError and tuple anchoring. + * STORM-708: CORS support for STORM UI. + * STORM-707: Client (Netty): improve logging to help troubleshooting connection woes + * STORM-704: Apply Travis CI to Apache Storm Project + * STORM-703: With hash key option for RedisMapState, only get values for keys in batch + * STORM-699: storm-jdbc should support custom insert queries. + * STORM-696: Single Namespace Test Launching + * STORM-694: java.lang.ClassNotFoundException: backtype.storm.daemon.common.SupervisorInfo + * STORM-693: KafkaBolt exception handling improvement. + * STORM-691: Add basic lookup / persist bolts + * STORM-690: Return Jedis into JedisPool with marking 'broken' if connection is broken + * STORM-689: SimpleACLAuthorizer should provide a way to restrict who can submit topologies. + * STORM-688: update Util to compile under JDK8 + * STORM-687: Storm UI does not display up to date information despite refreshes in IE + * STORM-685: wrong output in log when committed offset is too far behind latest offset + * STORM-684: In RichSpoutBatchExecutor: underlying spout is not closed when emitter is closed + * STORM-683: Make false in a conf really evaluate to false in clojure. + * STORM-682: supervisor should handle worker state corruption gracefully. + * STORM-681: Auto insert license header with genthrift.sh + * STORM-675: Allow users to have storm-env.sh under config dir to set custom JAVA_HOME and other env variables. + * STORM-673: Typo 'deamon' in security documentation + * STORM-672: Typo in Trident documentation example + * STORM-670: restore java 1.6 compatibility (storm-kafka) + * STORM-669: Replace links with ones to latest api document + * STORM-667: Incorrect capitalization "SHell" in Multilang-protocol.md + * STORM-663: Create javadocs for BoltDeclarer + * STORM-659: return grep matches each on its own line. + * STORM-657: make the shutdown-worker sleep time before kill -9 configurable + * STORM-656: Document "external" modules and "Committer Sponsors" + * STORM-651: improvements to storm.cmd + * STORM-641: Add total number of topologies to api/v1/cluster/summary. + * STORM-640: Storm UI vulnerable to poodle attack. + * STORM-637: Integrate PartialKeyGrouping into storm API + * STORM-636: Faster, optional retrieval of last component error + * STORM-635: logviewer returns 404 if storm_home/logs is a symlinked dir. + * STORM-634: Storm serialization changed to thrift to support rolling upgrade. + * STORM-632: New grouping for better load balancing + * STORM-630: Support for Clojure 1.6.0 + * STORM-629: Place Link to Source Code Repository on Webpage + * STORM-627: Storm-hbase configuration error. + * STORM-626: Add script to print out the merge command for a given pull request. + * STORM-625: Don't leak netty clients when worker moves or reuse netty client. + * STORM-623: Generate latest javadocs + * STORM-620: Duplicate maven plugin declaration + * STORM-616: Storm JDBC Connector. + * STORM-615: Add REST API to upload topology. + * STORM-613: Fix wrong getOffset return value + * STORM-612: Update the contact address in configure.ac + * STORM-611: Remove extra "break"s + * STORM-610: Check the return value of fts_close() + * STORM-609: Add storm-redis to storm external + * STORM-608: Storm UI CSRF escape characters not work correctly. + * STORM-607: storm-hbase HBaseMapState should support user to customize the hbase-key & hbase-qualifier + * STORM-603: Log errors when required kafka params are missing + * STORM-601: Make jira-github-join ignore case. + * STORM-600: upgrade jacoco plugin to support jdk8 + * STORM-599: Use nimbus's cached heartbeats rather than fetching again from ZK + * STORM-596: remove config topology.receiver.buffer.size + * STORM-595: storm-hdfs can only work with sequence files that use Writables. + * STORM-586: Trident kafka spout fails instead of updating offset when kafka offset is out of range. + * STORM-585: Performance issue in none grouping + * STORM-583: Add Microsoft Azure Event Hub spout implementations + * STORM-578: Calls to submit-mocked-assignment in supervisor-test use invalid executor-id format + * STORM-577: long time launch worker will block supervisor heartbeat + * STORM-575: Ability to specify Jetty host to bind to + * STORM-572: Storm UI 'favicon.ico' + * STORM-572: Allow Users to pass TEST-TIMEOUT-MS for java + * STORM-571: upgrade clj-time. + * STORM-570: Switch from tablesorter to datatables jquery plugin. + * STORM-569: Add Conf for bolt's outgoing overflow-buffer. + * STORM-567: Move Storm Documentation/Website from SVN to git + * STORM-565: Fix NPE when topology.groups is null. + * STORM-563: Kafka Spout doesn't pick up from the beginning of the queue unless forceFromStart specified. + * STORM-561: Add flux as an external module + * STORM-557: High Quality Images for presentations + * STORM-554: the type of first param "topology" should be ^StormTopology not ^TopologyContext + * STORM-552: Add netty socket backlog config + * STORM-548: Receive Thread Shutdown hook should connect to local hostname but not Localhost + * STORM-541: Build produces maven warnings + * STORM-539: Storm Hive Connector. + * STORM-533: Add in client and server IConnection metrics. + * STORM-527: update worker.clj -- delete "missing-tasks" checking + * STORM-525: Add time sorting function to the 2nd col of bolt exec table + * STORM-512: KafkaBolt doesn't handle ticks properly + * STORM-505: Fix debug string construction + * STORM-495: KafkaSpout retries with exponential backoff + * STORM-487: Remove storm.cmd, no need to duplicate work python runs on windows too. + * STORM-483: provide dedicated directories for classpath extension + * STORM-456: Storm UI: cannot navigate to topology page when name contains spaces. + * STORM-446: Allow superusers to impersonate other users in secure mode. + * STORM-444: Add AutoHDFS like credential fetching for HBase + * STORM-442: multilang ShellBolt/ShellSpout die() can be hang when Exception happened + * STORM-441: Remove bootstrap macro from Clojure codebase + * STORM-410: Add groups support to log-viewer + * STORM-400: Thrift upgrade to thrift-0.9.2 + * STORM-329: fix cascading Storm failure by improving reconnection strategy and buffering messages (thanks tedxia) + * STORM-322: Windows script do not handle spaces in JAVA_HOME path + * STORM-248: cluster.xml location is hardcoded for workers + * STORM-243: Record version and revision information in builds + * STORM-188: Allow user to specifiy full configuration path when running storm command + * STORM-130: Supervisor getting killed due to java.io.FileNotFoundException: File '../stormconf.ser' does not exist. http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_sass/_syntax-highlighting.scss ---------------------------------------------------------------------- diff --git a/_sass/_syntax-highlighting.scss b/_sass/_syntax-highlighting.scss new file mode 100644 index 0000000..a0d2404 --- /dev/null +++ b/_sass/_syntax-highlighting.scss @@ -0,0 +1,70 @@ +%vertical-rhythm { + margin-bottom: $spacing-unit / 2; +} +/** + * Syntax highlighting styles + */ +.highlight { + background: #fff; + @extend %vertical-rhythm; + + .c { color: #998; font-style: italic } // Comment + .err { color: #a61717; background-color: #e3d2d2 } // Error + .k { font-weight: bold } // Keyword + .o { font-weight: bold } // Operator + .cm { color: #998; font-style: italic } // Comment.Multiline + .cp { color: #999; font-weight: bold } // Comment.Preproc + .c1 { color: #998; font-style: italic } // Comment.Single + .cs { color: #999; font-weight: bold; font-style: italic } // Comment.Special + .gd { color: #000; background-color: #fdd } // Generic.Deleted + .gd .x { color: #000; background-color: #faa } // Generic.Deleted.Specific + .ge { font-style: italic } // Generic.Emph + .gr { color: #a00 } // Generic.Error + .gh { color: #999 } // Generic.Heading + .gi { color: #000; background-color: #dfd } // Generic.Inserted + .gi .x { color: #000; background-color: #afa } // Generic.Inserted.Specific + .go { color: #888 } // Generic.Output + .gp { color: #555 } // Generic.Prompt + .gs { font-weight: bold } // Generic.Strong + .gu { color: #aaa } // Generic.Subheading + .gt { color: #a00 } // Generic.Traceback + .kc { font-weight: bold } // Keyword.Constant + .kd { font-weight: bold } // Keyword.Declaration + .kp { font-weight: bold } // Keyword.Pseudo + .kr { font-weight: bold } // Keyword.Reserved + .kt { color: #458; font-weight: bold } // Keyword.Type + .m { color: #099 } // Literal.Number + .s { color: #d14 } // Literal.String + .na { color: #008080 } // Name.Attribute + .nb { color: #0086B3 } // Name.Builtin + .nc { color: #458; font-weight: bold } // Name.Class + .no { color: #008080 } // Name.Constant + .ni { color: #800080 } // Name.Entity + .ne { color: #900; font-weight: bold } // Name.Exception + .nf { color: #900; font-weight: bold } // Name.Function + .nn { color: #555 } // Name.Namespace + .nt { color: #000080 } // Name.Tag + .nv { color: #008080 } // Name.Variable + .ow { font-weight: bold } // Operator.Word + .w { color: #bbb } // Text.Whitespace + .mf { color: #099 } // Literal.Number.Float + .mh { color: #099 } // Literal.Number.Hex + .mi { color: #099 } // Literal.Number.Integer + .mo { color: #099 } // Literal.Number.Oct + .sb { color: #d14 } // Literal.String.Backtick + .sc { color: #d14 } // Literal.String.Char + .sd { color: #d14 } // Literal.String.Doc + .s2 { color: #d14 } // Literal.String.Double + .se { color: #d14 } // Literal.String.Escape + .sh { color: #d14 } // Literal.String.Heredoc + .si { color: #d14 } // Literal.String.Interpol + .sx { color: #d14 } // Literal.String.Other + .sr { color: #009926 } // Literal.String.Regex + .s1 { color: #d14 } // Literal.String.Single + .ss { color: #990073 } // Literal.String.Symbol + .bp { color: #999 } // Name.Builtin.Pseudo + .vc { color: #008080 } // Name.Variable.Class + .vg { color: #008080 } // Name.Variable.Global + .vi { color: #008080 } // Name.Variable.Instance + .il { color: #099 } // Literal.Number.Integer.Long +} http://git-wip-us.apache.org/repos/asf/storm/blob/1d09012e/_site/2012/08/02/storm080-released.html ---------------------------------------------------------------------- diff --git a/_site/2012/08/02/storm080-released.html b/_site/2012/08/02/storm080-released.html new file mode 100644 index 0000000..c670228 --- /dev/null +++ b/_site/2012/08/02/storm080-released.html @@ -0,0 +1,345 @@ +<!DOCTYPE html> +<html> + + <head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> + <link rel="icon" href="/favicon.ico" type="image/x-icon"> + + <title>Storm 0.8.0 and Trident released</title> + + <!-- Bootstrap core CSS --> + <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> + <!-- Bootstrap theme --> + <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> + + <!-- Custom styles for this template --> + <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css";> + <link href="/css/style.css" rel="stylesheet"> + <link href="/assets/css/owl.theme.css" rel="stylesheet"> + <link href="/assets/css/owl.carousel.css" rel="stylesheet"> + <script type="text/javascript" src="/assets/js/jquery.min.js"></script> + <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> + <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> + <script type="text/javascript" src="/assets/js/storm.js"></script> + <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> + <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> + + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> + <![endif]--> + </head> + + + + <body> + + <header> + <div class="container-fluid"> + <div class="row"> + <div class="col-md-10"> + <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> + </div> + <div class="col-md-2"> + <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> + </div> + </div> + </div> +</header> +<!--Header End--> +<!--Navigation Begin--> +<div class="navbar" role="banner"> + <div class="container-fluid"> + <div class="navbar-header"> + <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + </div> + <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> + <ul class="nav navbar-nav"> + <li><a href="/index.html" id="home">Home</a></li> + <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> + <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> + <li><a href="/documentation.html" id="documentation">Documentation</a></li> + <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Contribute <b class="caret"></b></a> + <ul class="dropdown-menu"> + <li><a href="/contribute/Contributing-to-Storm.html">Getting Started</a></li> + <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> + </ul> + </li> + <li><a href="/2015/06/15/storm0100-beta-released.html" id="news">News</a></li> + </ul> + </nav> + </div> +</div> + + + + <div class="container-fluid"> + <div class="row"> + <div class="col-md-12"> + <div class="row"> + <div class="col-md-3"> + <ul class="news" id="news-list"> + + <li><a href="/2015/06/15/storm0100-beta-released.html">Storm 0.10.0 Beta Released</a></li> + + <li><a href="/2015/06/04/storm095-released.html">Storm 0.9.5 released</a></li> + + <li><a href="/2015/03/25/storm094-released.html">Storm 0.9.4 released</a></li> + + <li><a href="/2014/11/25/storm093-released.html">Storm 0.9.3 released</a></li> + + <li><a href="/2014/10/20/storm093-release-candidate.html">Storm 0.9.3 release candidate 1 available</a></li> + + <li><a href="/2014/06/25/storm092-released.html">Storm 0.9.2 released</a></li> + + <li><a href="/2014/06/17/contest-results.html">Storm Logo Contest Results</a></li> + + <li><a href="/2014/05/27/round1-results.html">Logo Contest - Round 1 Results</a></li> + + <li><a href="/2014/04/29/logo-jlee3.html">Logo Entry No. 11 - Jennifer Lee</a></li> + + <li><a href="/2014/04/29/logo-jlee2.html">Logo Entry No. 10 - Jennifer Lee</a></li> + + <li><a href="/2014/04/29/logo-jlee1.html">Logo Entry No. 9 - Jennifer Lee</a></li> + + <li><a href="/2014/04/27/logo-sasili.html">Logo Entry No. 8 - Stefano Asili</a></li> + + <li><a href="/2014/04/27/logo-cboustead.html">Logo Entry No. 7 - Calum Boustead</a></li> + + <li><a href="/2014/04/23/logo-abartos.html">Logo Entry No. 6 - Alec Bartos</a></li> + + <li><a href="/2014/04/22/logo-zsayari.html">Logo Entry No. 5 - Ziba Sayari</a></li> + + <li><a href="/2014/04/21/logo-rmarshall.html">Logo Entry No. 4 - Richard Brownlie-Marshall</a></li> + + <li><a href="/2014/04/19/logo-ssuleman.html">Logo Entry No. 3- Shaan Shiv Suleman</a></li> + + <li><a href="/2014/04/17/logo-squinones.html">Logo Entry No. 2 - Samuel Quiñones</a></li> + + <li><a href="/2014/04/17/logo-pforrest.html">Logo Entry No. 1 - Patricia Forrest</a></li> + + <li><a href="/2014/04/10/storm-logo-contest.html">Apache Storm Logo Contest</a></li> + + <li><a href="/2013/12/08/storm090-released.html">Storm 0.9.0 Released</a></li> + + <li><a href="/2013/01/11/storm082-released.html">Storm 0.8.2 released</a></li> + + <li><a href="/2012/09/06/storm081-released.html">Storm 0.8.1 released</a></li> + + <li><a href="/2012/08/02/storm080-released.html">Storm 0.8.0 and Trident released</a></li> + + </ul> + </div> + <div class="col-md-9" id="news-content"> + <h1 class="page-title"> + Storm 0.8.0 and Trident released + </h1> + + <div class="row" style="margin: -15px;"> + <div class="col-md-12"> + <p class="text-muted credit pull-left">Posted on Aug 2, 2012 by Nathan Marz</p> + <div class="pull-right"> + <a + href="https://twitter.com/share"; + class="twitter-share-button" + data-count=none + >Tweet</a> + <script> !function(d,s,id){ + var js, + fjs=d.getElementsByTagName(s)[0], + p=/^http:/.test(d.location)?'http':'https'; + if(!d.getElementById(id)){ + js=d.createElement(s); + js.id=id; + js.src=p+'://platform.twitter.com/widgets.js'; + fjs.parentNode.insertBefore(js,fjs); + } + }(document, 'script', 'twitter-wjs'); + </script> + </div> + </div> + </div> + <div> + <p>I'm happy to announce the release of Storm 0.8.0. This is a <em>huge</em> release. Thanks to everyone who tested out the dev release and helped find and fix issues. And thanks to everyone who contributed pull requests. There's one big new thing available in the release: Trident. You may have heard me hint about Trident before, and now it's finally public. </p> + +<p>Trident is a higher level abstraction for Storm. It allows you to seamlessly mix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar - Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies. Trident is bundled with Storm, and you can read documentation about it on <a href="https://github.com/apache/incubator-storm/wiki/Documentation";>this page</a> (start with "Trident tutorial").</p> + +<p>Trident supersedes both LinearDRPCTopologyBuilder and transactional topologies, both of which will be deprecated soon and eventually removed from the codebase. </p> + +<p>Here are the other highlights of this release: </p> + +<h2 id="executors">Executors</h2> + +<p>Prior to Storm 0.8.0, a running topology consisted of some number of workers and some number of tasks that ran on those workers. In the old model, worker = process and task = thread. Storm 0.8.0 changes this model by introducing executors. In this model, a worker = process, an executor = thread, and one executor runs many tasks from the same spout/bolt. </p> + +<p>The reason for the change is that the old model complected the semantics of the topology with its physical execution. For example, if you had a bolt with 4 tasks doing a fields grouping on some stream, in order to maintain the semantics of the fields grouping (that the same value always goes to the same task id for that bolt), the number of tasks for that bolt needs to be fixed for the lifetime of the topology, and since task = thread, the number of threads for that bolt is fixed for the lifetime of the topology. In the new model, the number of threads for a bolt is disassociated from the number of tasks, meaning you can change the number of threads for a spout/bolt dynamically without affecting semantics. Similarly, if you're keeping large amounts of state in your bolts, and you want to increase the parallelism of the bolt without having to repartition the state, you can do this with the new executors. </p> + +<p>At the API level, the "parallelism_hint" now specifies the initial number of executors for that bolt. You can specify the number of tasks using the TOPOLOGY_TASKS component config. For example:</p> + +<p><code> +builder.setBolt(new MyBolt(), 3).setNumTasks(128).shuffleGrouping("spout"); +</code></p> + +<p>This sets the initial number of executors to 3 and the number of tasks to 128. If you don't specify the number of tasks for a component, it will be fixed to the initial number of executors for the lifetime of the topology. </p> + +<p>Finally, you can change the number of workers and/or number of executors for components using the "storm rebalance" command. The following command changes the number of workers for the "demo" topology to 3, the number of executors for the "myspout" component to 5, and the number of executors for the "mybolt" component to 1: </p> + +<p><code> +storm rebalance demo -n 3 -e myspout=5 -e mybolt=1 +</code></p> + +<h2 id="pluggable-scheduler">Pluggable scheduler</h2> + +<p>You can now implement your own scheduler to replace the default scheduler to assign executors to workers. You configure the class to use using the "storm.scheduler" config in your storm.yaml, and your scheduler must implement <a href="https://github.com/apache/incubator-storm/blob/master/src/jvm/backtype/storm/scheduler/IScheduler.java";>this interface</a>.</p> + +<h2 id="throughput-improvements">Throughput improvements</h2> + +<p>The internals of Storm have been rearchitected for extremely significant performance gains. I'm seeing throughput increases of anywhere from 5-10x of what it was before. I've benchmarked Storm at 1M tuples / sec / node on an internal Twitter cluster. </p> + +<p>The key changes made were: </p> + +<p>a) Replacing all the internal in-memory queuing with the LMAX Disruptor +b) Doing intelligent auto-batching of processing so that the consumers can keep up with the producers </p> + +<p>Here are the configs which affect how buffering/batching is done: </p> + +<p>topology.executor.receive.buffer.size +topology.executor.send.buffer.size +topology.receiver.buffer.size +topology.transfer.buffer.size </p> + +<p>These may require some tweaking to optimize your topologies, but most likely the default values will work fine for you out of the box. </p> + +<h2 id="decreased-zookeeper-load-/-increased-storm-ui-performance">Decreased Zookeeper load / increased Storm UI performance</h2> + +<p>Storm sends significantly less traffic to Zookeeper now (on the order of 10x less). Since it also uses so many fewer znodes to store state, the UI is significantly faster as well. </p> + +<h2 id="abstractions-for-shared-resources">Abstractions for shared resources</h2> + +<p>The TopologyContext has methods "getTaskData", "getExecutorData", and "getResource" for sharing resources at the task level, executor level, or worker level respectively. Currently you can't set any worker resources, but this is in development. Storm currently provides a shared ExecutorService worker resource (via "getSharedExecutor" method) that can be used for launching background tasks on a shared thread pool. </p> + +<h2 id="tick-tuples">Tick tuples</h2> + +<p>It's common to require a bolt to "do something" at a fixed interval, like flush writes to a database. Many people have been using variants of a ClockSpout to send these ticks. The problem with a ClockSpout is that you can't internalize the need for ticks within your bolt, so if you forget to set up your bolt correctly within your topology it won't work correctly. 0.8.0 introduces a new "tick tuple" config that lets you specify the frequency at which you want to receive tick tuples via the "topology.tick.tuple.freq.secs" component- specific config, and then your bolt will receive a tuple from the __system component and __tick stream at that frequency. </p> + +<h2 id="improved-storm-ui">Improved Storm UI</h2> + +<p>The Storm UI now has a button for showing/hiding the "system stats" (tuples sent on streams other than ones you've defined, like acker streams), making it easier to digest what your topology is doing.</p> + +<p>Here's the full changelog for Storm 0.8.0: </p> + +<ul> +<li>Added Trident, the new high-level abstraction for intermixing high throughput, stateful stream processing with low-latency distributed querying </li> +<li>Added executor abstraction between workers and tasks. Workers = processes, executors = threads that run many tasks from the same spout or bolt. </li> +<li>Pluggable scheduler (thanks xumingming) </li> +<li>Eliminate explicit storage of task->component in Zookeeper </li> +<li>Number of workers can be dynamically changed at runtime through rebalance command and -n switch </li> +<li>Number of executors for a component can be dynamically changed at runtime through rebalance command and -e switch (multiple -e switches allowed) </li> +<li>Use worker heartbeats instead of task heartbeats (thanks xumingming) </li> +<li>UI performance for topologies with many executors/tasks much faster due to optimized usage of Zookeeper (10x improvement) </li> +<li>Added button to show/hide system stats (e.g., acker component and stream stats) from the Storm UI (thanks xumingming) </li> +<li>Stats are tracked on a per-executor basis instead of per-task basis </li> +<li>Major optimization for unreliable spouts and unanchored tuples (will use far less CPU) </li> +<li>Revamped internals of Storm to use LMAX disruptor for internal queuing. Dramatic reductions in contention and CPU usage. </li> +<li>Numerous micro-optimizations all throughout the codebase to reduce CPU usage. </li> +<li>Optimized internals of Storm to use much fewer threads - two fewer threads per spout and one fewer thread per acker. </li> +<li>Removed error method from task hooks (to be re-added at a later time) </li> +<li>Validate that subscriptions come from valid components and streams, and if it's a field grouping that the schema is correct (thanks xumingming) </li> +<li>MemoryTransactionalSpout now works in cluster mode </li> +<li>Only track errors on a component by component basis to reduce the amount stored in zookeeper (to speed up UI). A side effect of this change is the removal of the task page in the UI. </li> +<li>Add TOPOLOGY-TICK-TUPLE-FREQ-SECS config to have Storm automatically send "tick" tuples to a bolt's execute method coming from the __system component and __tick stream at the configured frequency. Meant to be used as a component-specific configuration. </li> +<li>Upgrade Kryo to v2.17 </li> +<li>Tuple is now an interface and is much cleaner. The Clojure DSL helpers have been moved to TupleImpl </li> +<li>Added shared worker resources. Storm provides a shared ExecutorService thread pool by default. The number of threads in the pool can be configured with topology.worker.shared.thread.pool.size </li> +<li>Improve CustomStreamGrouping interface to make it more flexible by providing more information </li> +<li>Enhanced INimbus interface to allow for forced schedulers and better integration with global scheduler </li> +<li>Added assigned method to ISupervisor so it knows exactly what's running and not running </li> +<li>Custom serializers can now have one of four constructors: (), (Kryo), (Class), or (Kryo, Class) </li> +<li>Disallow ":", ".", and "\" from topology names </li> +<li>Errors in multilang subprocesses that go to stderr will be captured and logged to the worker logs (thanks vinodc) </li> +<li>Workers detect and warn for missing outbound connections from assignment, drop messages for which there's no outbound connection </li> +<li>Zookeeper connection timeout is now configurable (via storm.zookeeper.connection.timeout config) </li> +<li>Storm is now less aggressive about halting process when there are Zookeeper errors, preferring to wait until client calls return exceptions. </li> +<li>Can configure Zookeeper authentication for Storm's Zookeeper clients via "storm.zookeeper.auth.scheme" and "storm.zookeeper.auth.payload" configs </li> +<li>Supervisors only download code for topologies assigned to them </li> +<li>Include task id information in task hooks (thanks velvia) </li> +<li>Use execvp to spawn daemons (replaces the python launcher process) (thanks ept) </li> +<li>Expanded INimbus/ISupervisor interfaces to provide more information (used in Storm/Mesos integration) </li> +<li>Bug fix: Realize task ids when worker heartbeats to supervisor. Some users were hitting deserialization problems here in very rare cases (thanks herberteuler) </li> +<li>Bug fix: Fix bug where a topology's status would get corrupted to true if nimbus is restarted while status is rebalancing </li> +</ul> + + </div> + </div> + </div> + </div> + </div> + </div> + <footer> + <div class="container-fluid"> + <div class="row"> + <div class="col-md-3"> + <div class="footer-widget"> + <h5>Meetups</h5> + <ul class="latest-news"> + <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/";>Sunnyvale, CA</a> <span class="small">(10 May 2015)</span></li> + <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/";>Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> + </ul> + </div> + </div> + <div class="col-md-3"> + <div class="footer-widget"> + <h5>About Storm</h5> + <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> + </div> + </div> + <div class="col-md-3"> + <div class="footer-widget"> + <h5>First Look</h5> + <ul class="footer-list"> + <li><a href="/documentation/Rationale.html">Rationale</a></li> + <li><a href="/tutorial.html">Tutorial</a></li> + <li><a href="/documentation/Setting-up-development-environment.html">Setting up development environment</a></li> + <li><a href="/documentation/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> + </ul> + </div> + </div> + <div class="col-md-3"> + <div class="footer-widget"> + <h5>Documentation</h5> + <ul class="footer-list"> + <li><a href="/doc-index.html">Index</a></li> + <li><a href="/documentation.html">Manual</a></li> + <li><a href="https://storm.apache.org/javadoc/apidocs/index.html";>Javadoc</a></li> + <li><a href="/documentation/FAQ.html">FAQ</a></li> + </ul> + </div> + </div> + </div> + <hr/> + <div class="row"> + <div class="col-md-12"> + <p align="center">Copyright © 2015 <a href="http://www.apache.org";>Apache Software Foundation</a>. All Rights Reserved. Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> + </div> + </div> + </div> +</footer> +<!--Footer End--> +<!-- Scroll to top --> +<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> + + </body> + +</html> +
