Hai Zhou UTC+8 created FLINK-7964:
-------------------------------------
Summary: Add Apache Kafka 1.0.0 connector
Key: FLINK-7964
URL: https://issues.apache.org/jira/browse/FLINK-7964
Project: Flink
Issue Type: Improvement
Components: Kafka Connector
Affects Versions: 1.4.0
Reporter: Hai Zhou UTC+8
Assignee: Hai Zhou UTC+8
Priority: Major
Fix For: 1.5.0
Kafka 1.0.0 is no mere bump of the version number. The Apache Kafka Project
Management Committee has packed a number of valuable enhancements into the
release. Here is a summary of a few of them:
* Since its introduction in version 0.10, the Streams API has become hugely
popular among Kafka users, including the likes of Pinterest, Rabobank, Zalando,
and The New York Times. In 1.0, the the API continues to evolve at a healthy
pace. To begin with, the builder API has been improved (KIP-120). A new API has
been added to expose the state of active tasks at runtime (KIP-130). The new
cogroup API makes it much easier to deal with partitioned aggregates with fewer
StateStores and fewer moving parts in your code (KIP-150). Debuggability gets
easier with enhancements to the print() and writeAsText() methods (KIP-160).
And if that’s not enough, check out KIP-138 and KIP-161 too. For more on
streams, check out the Apache Kafka Streams documentation, including some
helpful new tutorial videos.
* Operating Kafka at scale requires that the system remain observable, and to
make that easier, we’ve made a number of improvements to metrics. These are too
many to summarize without becoming tedious, but Connect metrics have been
significantly improved (KIP-196), a litany of new health check metrics are now
exposed (KIP-188), and we now have a global topic and partition count
(KIP-168). Check out KIP-164 and KIP-187 for even more.
* We now support Java 9, leading, among other things, to significantly faster
TLS and CRC32C implementations. Over-the-wire encryption will be faster now,
which will keep Kafka fast and compute costs low when encryption is enabled.
* In keeping with the security theme, KIP-152 cleans up the error handling on
Simple Authentication Security Layer (SASL) authentication attempts.
Previously, some authentication error conditions were indistinguishable from
broker failures and were not logged in a clear way. This is cleaner now.
* Kafka can now tolerate disk failures better. Historically, JBOD storage
configurations have not been recommended, but the architecture has nevertheless
been tempting: after all, why not rely on Kafka’s own replication mechanism to
protect against storage failure rather than using RAID? With KIP-112, Kafka now
handles disk failure more gracefully. A single disk failure in a JBOD broker
will not bring the entire broker down; rather, the broker will continue serving
any log files that remain on functioning disks.
* Since release 0.11.0, the idempotent producer (which is the producer used in
the presence of a transaction, which of course is the producer we use for
exactly-once processing) required max.in.flight.requests.per.connection to be
equal to one. As anyone who has written or tested a wire protocol can attest,
this put an upper bound on throughput. Thanks to KAFKA-5949, this can now be as
large as five, relaxing the throughput constraint quite a bit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
