Thanks Reuven !
I would like to emphasize on some highlights in 2.2.0 release:
- New IOs have been introduced:
* TikaIO leveraging Apache Tika, allowing the deal with a lot of different
data formats
* RedisIO to read and write key/value pairs from a Redis server. This IO will
be soon extended to Redis PubSub.
* FileIO provides transforms for working with files (raw). Especially, it
provides matching file patterns and read on patterns. It can be easily extended
for a specific format (like we do in AvroIO or TextIO now).
* SolrIO to interact with Apache Solr (Lucene)
- On the other hand, improvements have been performed on existing IOs:
* We started to introduce readAll pattern in IOs (AvroIO, TextIO, JdbcIO,
...), allowing to pass "request" arguments via an input PCollection.
* ElasticsearchIO has an improved support of different Elasticsearch version
(including Elasticsearch 5.x). It also now supports SSL/TLS.
* HBaseIO is now able to do dynamic work rebalancing
* KinesisIO uses a more accurate watermark (based on
approximateArrivalTimestamp)
* TextIO now supports custom delimiter and like AvroIO, supports the readAll
pattern,
* Performance improvements on JdbcIO when it has to read lot of rows
* Kafka write supports Exactly-Once pattern (introduce in Kafka 0.11.x)
- A new DSL has been introduced: the SQL DSL !
We are now focus on 2.3.0 release with new improvements and features !
Stay tuned !
JB on behalf of the Apache Beam community.
On 12/02/2017 11:40 PM, Reuven Lax wrote:
The Apache Beam community is pleased to announce the availability of the
2.2.0 release.
This release adds support for generic file sources and sinks (beyond TextIO and
AvroIO) using FileIO, including support for dynamic filenames using readAll;
this allows streaming pipelines to now read from files by continuously
monitoring a directory for new filw. Many other IOs are improved, notably
including exactly-once support for the Kafka sink. Initial support for BEAM-SQL
is also included in this release. For a more-complete list of major changes in
the release, please refer to the release notes [2].
The 2.2.0 release is now the recommended version; we encourage everyone to
upgrade from any earlier releases.
We’d like to invite everyone to try out Apache Beam today and consider
joining our vibrant community. We welcome feedback, contribution and
participation through our mailing lists, issue tracker, pull requests, and
events.
- Reuven Lax, on behalf of the Apache Beam community.
[1] https://beam.apache.org/get-started/downloads/
[2]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
