[
https://issues.apache.org/jira/browse/FLINK-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-7964:
------------------------------------
Summary: Add Apache Kafka 1.0/1.1 connector (was: Add Apache Kafka 1.0
connector)
> Add Apache Kafka 1.0/1.1 connector
> ----------------------------------
>
> Key: FLINK-7964
> URL: https://issues.apache.org/jira/browse/FLINK-7964
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Affects Versions: 1.4.0
> Reporter: Hai Zhou
> Assignee: Hai Zhou
> Priority: Major
> Fix For: 1.7.0
>
>
> Kafka 1.0.0 is no mere bump of the version number. The Apache Kafka Project
> Management Committee has packed a number of valuable enhancements into the
> release. Here is a summary of a few of them:
> * Since its introduction in version 0.10, the Streams API has become hugely
> popular among Kafka users, including the likes of Pinterest, Rabobank,
> Zalando, and The New York Times. In 1.0, the the API continues to evolve at a
> healthy pace. To begin with, the builder API has been improved (KIP-120). A
> new API has been added to expose the state of active tasks at runtime
> (KIP-130). The new cogroup API makes it much easier to deal with partitioned
> aggregates with fewer StateStores and fewer moving parts in your code
> (KIP-150). Debuggability gets easier with enhancements to the print() and
> writeAsText() methods (KIP-160). And if that’s not enough, check out KIP-138
> and KIP-161 too. For more on streams, check out the Apache Kafka Streams
> documentation, including some helpful new tutorial videos.
> * Operating Kafka at scale requires that the system remain observable, and to
> make that easier, we’ve made a number of improvements to metrics. These are
> too many to summarize without becoming tedious, but Connect metrics have been
> significantly improved (KIP-196), a litany of new health check metrics are
> now exposed (KIP-188), and we now have a global topic and partition count
> (KIP-168). Check out KIP-164 and KIP-187 for even more.
> * We now support Java 9, leading, among other things, to significantly faster
> TLS and CRC32C implementations. Over-the-wire encryption will be faster now,
> which will keep Kafka fast and compute costs low when encryption is enabled.
> * In keeping with the security theme, KIP-152 cleans up the error handling on
> Simple Authentication Security Layer (SASL) authentication attempts.
> Previously, some authentication error conditions were indistinguishable from
> broker failures and were not logged in a clear way. This is cleaner now.
> * Kafka can now tolerate disk failures better. Historically, JBOD storage
> configurations have not been recommended, but the architecture has
> nevertheless been tempting: after all, why not rely on Kafka’s own
> replication mechanism to protect against storage failure rather than using
> RAID? With KIP-112, Kafka now handles disk failure more gracefully. A single
> disk failure in a JBOD broker will not bring the entire broker down; rather,
> the broker will continue serving any log files that remain on functioning
> disks.
> * Since release 0.11.0, the idempotent producer (which is the producer used
> in the presence of a transaction, which of course is the producer we use for
> exactly-once processing) required max.in.flight.requests.per.connection to be
> equal to one. As anyone who has written or tested a wire protocol can attest,
> this put an upper bound on throughput. Thanks to KAFKA-5949, this can now be
> as large as five, relaxing the throughput constraint quite a bit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
