[jira] [Created] (KAFKA-6066) Use of SimpleDateFormat in RocksDBWindowStore may not be Threadsafe

[Srikanth Sundarrajan (JIRA)]

Srikanth Sundarrajan created KAFKA-6066:
---

 Summary: Use of SimpleDateFormat in RocksDBWindowStore may not be 
Threadsafe
 Key: KAFKA-6066
 URL: https://issues.apache.org/jira/browse/KAFKA-6066
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Srikanth Sundarrajan
Priority: Minor


Currently SimpleDateFormat is used to construct segmentId from segmentName and 
vice-versa. This however may not be thread safe if WindowStore is accessed by 
more than one SteamTask/thread concurrently. 
Ref: *org.apache.kafka.streams.state.internals.RocksDBWindowStore#formatter*



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

[Becket Qin]

Hi Paolo,

Thanks for the KIP and sorry for being late on the thread. I am wondering
what is the KafkaFuture returned by all() call? Should it be a
Map instead?

Thanks,

Jiangjie (Becket) QIn

On Thu, Sep 28, 2017 at 3:48 AM, Paolo Patierno  wrote:

> Hi,
>
>
> maybe we want to start without the delete records policy for now waiting
> for a real needs. So I'm removing it from the KIP.
>
> I hope for more comments on this KIP-204 so that we can start a vote on
> Monday.
>
>
> Thanks.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Paolo Patierno 
> Sent: Thursday, September 28, 2017 5:56 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi,
>
>
> I have just updated the KIP-204 description with the new
> TopicDeletionPolicy suggested by the KIP-201.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Paolo Patierno 
> Sent: Tuesday, September 26, 2017 4:57 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Tom,
>
> as I said in the KIP-201 discussion I'm ok with having a unique
> DeleteTopicPolicy but then it could be useful having more information then
> just the topic name; partitions and offset for messages deletion could be
> useful for a fine grained use cases.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Tom Bentley 
> Sent: Tuesday, September 26, 2017 4:32 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> new Admin Client API
>
> Hi Paolo,
>
> I guess a RecordDeletionPolicy should work at the partition level, whereas
> the TopicDeletionPolicy should work at the topic level. But then we run
> into a similar situation as described in the motivation for KIP-201, where
> the administrator might have to write+configure two policies in order to
> express their intended rules. For example, it's no good preventing people
> from deleting topics if they can delete all the messages in those topics,
> or vice versa.
>
> On that reasoning, perhaps there should be a single policy interface
> covering topic deletion and message deletion. Alternatively, the topic
> deletion API could also invoke the record deletion policy (before the topic
> deletion policy I mean). But the former would be more consistent with
> what's proposed in KIP-201.
>
> Wdyt? I can add this to KIP-201 if you want.
>
> Cheers,
>
> Tom
>
>
>
>
>
> On 26 September 2017 at 17:01, Paolo Patierno  wrote:
>
> > Hi Tom,
> >
> > I think that we could live with the current authorizer based on delete
> > topic (for both, deleting messages and topic as a whole) but then the
> > RecordsDeletePolicy could be even more fine grained giving the
> possibility
> > to avoid deleting messages for specific partitions (inside the topic) and
> > starting from a specific offset.
> >
> > I could think on some users solutions where they know exactly what the
> > partitions means inside of a specific topic (because they are using a
> > custom partitioner on the producer side) so they know what kind of
> messages
> > are inside a partition allowing to delete them but not the other.
> >
> > In such a policy a user could also check the timestamp related to the
> > offset for allowing or not deletion on time base.
> >
> >
> > Wdyt ?
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Tom Bentley 
> > Sent: Tuesday, September 26, 2017 2:55 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-204 : adding records deletion operation to the
> > new Admin Client API
> >
> > Hi Edoardo and 

[jira] [Created] (KAFKA-6065) Add zookeeper metrics to ZookeeperClient as in KIP-188

[Onur Karaman (JIRA)]

Onur Karaman created KAFKA-6065:
---

 Summary: Add zookeeper metrics to ZookeeperClient as in KIP-188
 Key: KAFKA-6065
 URL: https://issues.apache.org/jira/browse/KAFKA-6065
 Project: Kafka
  Issue Type: Sub-task
Reporter: Onur Karaman


Among other things, KIP-188 added latency metrics to ZkUtils. We should add the 
same metrics to ZookeeperClient.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk7 #2894

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: update exception message for KIP-120

[jason] MINOR: A few javadoc fixes

[wangguoz] MINOR: add equals to SessionWindows

--
[...truncated 1.82 MB...]
org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin STARTED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableLeftJoin PASSED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin STARTED

org.apache.kafka.streams.integration.GlobalKTableIntegrationTest > 
shouldKStreamGlobalKTableJoin PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable STARTED

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

[Richard Yu]

As Guozhang Wang mentioned earlier, we want to mirror the structure of
similar Store class (namely KTable). The WindowedStore class might be
unique in itself as it uses fetch() methods, but in my opinion, uniformity
should be better suited for simplicity.

On Mon, Oct 16, 2017 at 11:54 AM, Xavier Léauté  wrote:

> Thank you Richard! Do you or Guozhang have any thoughts on my suggestions
> to use fetchAll() and fetchAll(timeFrom, timeTo) and reserve the "range"
> keyword for when we query a specific range of keys?
>
> Xavier
>
> On Sat, Oct 14, 2017 at 2:32 PM Richard Yu 
> wrote:
>
> > Thanks for the clarifications, Xavier.
> > I have removed most of the methods except for keys() and all() which has
> > been renamed to Guozhang Wang's suggestions.
> >
> > Hope this helps.
> >
> > On Fri, Oct 13, 2017 at 3:28 PM, Xavier Léauté 
> > wrote:
> >
> > > Thanks for the KIP Richard, this is a very useful addition!
> > >
> > > As far as the API changes, I just have a few comments on the methods
> that
> > > don't seem directly related to the KIP title, and naming of course :).
> > > On the implementation, see my notes further down that will hopefully
> > > clarify a few things.
> > >
> > > Regarding the "bonus" methods:
> > > I agree with Guozhang that the KIP lacks proper motivation for adding
> the
> > > min, max, and allLatest methods.
> > > It is also not clear to me what min and max would really mean, what
> > > ordering do we refer to here? Are we first ordering by time, then key,
> or
> > > first by key, then time?
> > > The allLatest method might be useful, but I don't really see how it
> would
> > > be used in practice if we have to scan the entire range of keys for all
> > the
> > > state stores, every single time.
> > >
> > > Maybe we could flesh the motivation behind those extra methods, but in
> > the
> > > interest of time, and moving the KIP forward it might make sense to
> file
> > a
> > > follow-up once we have more concrete use-cases.
> > >
> > > On naming:
> > > I also agree with Guozhang that "keys()" should be renamed. It feels a
> > bit
> > > of a misnomer, since it not only returns keys, but also the values.
> > >
> > > As far as what to rename it to, I would argue we already have some
> > > discrepancy between key-value stores using range() vs. window stores
> > using
> > > fetch().
> > > I assume we called the window method "fetch" instead of "get" because
> you
> > > might get back more than one window for the requested key.
> > >
> > > If we wanted to make things consistent with both existing key-value
> store
> > > naming and window store naming, we could do the following:
> > > Decide that "all" always refers to the entire range of keys,
> independent
> > of
> > > the window and similarly "range" always refers to a particular range of
> > > keys, irrespective of the window.
> > > We can then prefix methods with "fetch" to indicate that more than one
> > > window may be returned for each key in the range.
> > >
> > > This would give us:
> > > - a new fetchAll() method for all the keys, which makes it clear that
> you
> > > might get back the same key in different windows
> > > - a new fetchAll(timeFrom, timeTo) method to get all the keys in a
> given
> > > time range, again with possibly more than one window per key
> > > - and we'd have to rename fetch(K,K,long, long) to fetchRange(K, K,
> long,
> > > long)  and deprecate the old one to indicate a range of keys
> > >
> > > One inconsistency I noted: the "Proposed Changes" section in your KIP
> > talks
> > > about a "range(timeFrom, timeTo)" method, I think you meant to refer to
> > the
> > > all(from, to) method, but I'm sure you'll fix that once we decide on
> > > naming.
> > >
> > > On the implementation side:
> > > You mentioned that caching and rocksdb store have very different
> > key/value
> > > structures, and while it appears to be that way on the surface, the
> > > structure between the two is actually very similar. Keys in the cache
> are
> > > prefixed with a segment ID to ensure the ordering in the cache stays
> > > consistent with the rocksdb implementation, which maintains multiple
> > > rocksdb instances, one for each segment. So we just "artificially"
> mirror
> > > the segment structure in the cache.
> > >
> > > The reason for keeping the ordering consistent is pretty simple: keep
> in
> > > mind that when we query a cached window store we are effectively
> querying
> > > both the cache and the persistent rocksdb store at the same time,
> merging
> > > results from both. To make that merge as painless as possible, we
> ensure
> > > the ordering is consistent when querying a range of keys in both
> stores.
> > >
> > > Also keep in mind CompositeReadonlyWindowStore, which wraps multiple
> > window
> > > stores within a topology.
> > >
> > > Hope this clarifies some of the less trivial parts of caching window
> > store.
> > >
> > > Cheers,
> > > Xavier

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

[Dong Lin]

Thanks for the KIP. +1 (non-binding)

On Wed, Oct 11, 2017 at 2:27 AM, Ted Yu  wrote:

> +1
>
> On Mon, Oct 2, 2017 at 10:51 PM, Paolo Patierno 
> wrote:
>
> > Hi all,
> >
> > I didn't see any further discussion around this KIP, so I'd like to start
> > the vote for it.
> >
> > Just for reference : https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-204+%3A+adding+records+
> > deletion+operation+to+the+new+Admin+Client+API
> >
> >
> > Thanks,
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Azure & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
>

Re: [VOTE] 1.0.0 RC1

[Ismael Juma]

If you don't use the default Scala version, you have to set the
SCALA_VERSION environment variable for the bin scripts to work.

Ismael

On 17 Oct 2017 1:30 am, "Vahid S Hashemian" 
wrote:

Hi Guozhang,

I'm not sure if this should be covered by "Java 9 support" in the RC note,
but when I try to build jars from source using Java 9 (./gradlew
-PscalaVersion=2.12 jar) even though the build reports as succeeded, it
doesn't seem to have been successful:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
Error: Could not find or load main class
org.apache.zookeeper.server.quorum.QuorumPeerMain
Caused by: java.lang.ClassNotFoundException:
org.apache.zookeeper.server.quorum.QuorumPeerMain

Please advise if I'm missing something.

Thanks.
--Vahid




From:   Guozhang Wang 
To: "dev@kafka.apache.org" ,
"us...@kafka.apache.org" , kafka-clients

Date:   10/13/2017 01:12 PM
Subject:[VOTE] 1.0.0 RC1



Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 1.0.0.

It's worth noting that starting in this version we are using a different
version protocol with three digits: *major.minor.bug-fix*

Any and all testing is welcome, but the following areas are worth
highlighting:

1. Client developers should verify that their clients can produce/consume
to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
2. Performance and stress testing. Heroku and LinkedIn have helped with
this in the past (and issues have been found and fixed).
3. End users can verify that their apps work correctly with the new
release.

This is a major version release of Apache Kafka. It includes 29 new KIPs.
See the release notes and release plan
(*https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_
confluence_pages_viewpage.action-3FpageId-3D71764913=
DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
tT9k0x5RvXtHEtLzp03BA1Y8DAgHzgCXD7UjqP7oiKE=
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
apache.org_confluence_pages_viewpage.action-3FpageId-
3D71764913=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
tT9k0x5RvXtHEtLzp03BA1Y8DAgHzgCXD7UjqP7oiKE=
>*)
for more details. A few feature highlights:

* Java 9 support with significantly faster TLS and CRC32C implementations
(KIP)
* JBOD improvements: disk failure only disables failed disk but not the
broker (KIP-112/KIP-113)
* Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
KIP-188, KIP-196)
* Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
and drop compatibility "Evolving" annotations

Release notes for the 1.0.0 release:
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
7Eguozhang_kafka-2D1.0.0-2Drc1_RELEASE-5FNOTES.html=
DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
xopSUD2TETEI5y8kxHM4P-jUdUKUIiUig2xVwabgDq8=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_RELEASE-5FNOTES.
html=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
xopSUD2TETEI5y8kxHM4P-jUdUKUIiUig2xVwabgDq8=
>*



*** Please download, test and vote by Tuesday, October 13, 8pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
apache.org_KEYS=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_
OWk2y2QfKYsXitTyAHHM=FfLcWlN8ODpZ2m1KliMfp35duIxif3FNnptY5-9JKWU=


* Release artifacts to be voted upon (source and binary):
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
7Eguozhang_kafka-2D1.0.0-2Drc1_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_
OWk2y2QfKYsXitTyAHHM=bcWIqj27_tkoj-fnEzcLdP8uGXyAt6gS9KUy12WF1FE=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_=DwIBaQ=jf_
iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=
VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=bcWIqj27_tkoj-
fnEzcLdP8uGXyAt6gS9KUy12WF1FE=
>*

* Maven artifacts to be voted upon:
https://urldefense.proofpoint.com/v2/url?u=https-3A__
repository.apache.org_content_groups_staging_org_apache_
kafka_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=dKi_J-
8X1TkZ83fa3hLkcO0qGcuYQ0lTxtK4o6ms5m0=


* Javadoc:
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
7Eguozhang_kafka-2D1.0.0-2Drc1_javadoc_=DwIBaQ=jf_
iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=

Jenkins build is back to normal : kafka-trunk-jdk9 #128

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #2893

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: a few web doc and javadoc fixes

--
[...truncated 1.82 MB...]
org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldRollSegments PASSED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldFindValuesWithinRange STARTED

org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStoreTest > 
shouldFindValuesWithinRange PASSED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowForUnknownKeyTypeForBuiltinTypes STARTED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowForUnknownKeyTypeForBuiltinTypes PASSED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowIfValueClassIsNullForBuiltinTypes STARTED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowIfValueClassIsNullForBuiltinTypes PASSED

org.apache.kafka.streams.state.StateSerdesTest > shouldThrowIfTopicNameIsNull 
STARTED

org.apache.kafka.streams.state.StateSerdesTest > shouldThrowIfTopicNameIsNull 
PASSED

org.apache.kafka.streams.state.StateSerdesTest > shouldThrowIfKeyClassIsNull 
STARTED

org.apache.kafka.streams.state.StateSerdesTest > shouldThrowIfKeyClassIsNull 
PASSED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldReturnSerdesForBuiltInKeyAndValueTypesForBuiltinTypes STARTED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldReturnSerdesForBuiltInKeyAndValueTypesForBuiltinTypes PASSED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowIfTopicNameIsNullForBuiltinTypes STARTED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowIfTopicNameIsNullForBuiltinTypes PASSED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowForUnknownValueTypeForBuiltinTypes STARTED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowForUnknownValueTypeForBuiltinTypes PASSED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowIfKeyClassIsNullForBuiltinTypes STARTED

org.apache.kafka.streams.state.StateSerdesTest > 
shouldThrowIfKeyClassIsNullForBuiltinTypes PASSED

org.apache.kafka.streams.state.StateSerdesTest > shouldThrowIfValueClassIsNull 
STARTED

org.apache.kafka.streams.state.StateSerdesTest > shouldThrowIfValueClassIsNull 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled 
STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Guozhang Wang]

Regarding #6 above, I'm still not clear why we would need `commit()` in
both ProcessorContext and RecordContext, could you elaborate a bit more?

To me `commit()` is really a processor context not a record context
logically: when you call that function, it means we would commit the state
of the whole task up to this processed record, not only that single record
itself.


Guozhang

On Mon, Oct 16, 2017 at 9:19 AM, Jeyhun Karimov 
wrote:

> Hi,
>
> Thanks for the feedback.
>
>
> 0. RichInitializer definition seems missing.
>
>
>
> - Fixed.
>
>
>  I'd suggest moving the key parameter in the RichValueXX and RichReducer
> > after the value parameters, as well as in the templates; e.g.
> > public interface RichValueJoiner {
> > VR apply(final V1 value1, final V2 value2, final K key, final
> > RecordContext
> > recordContext);
> > }
>
>
>
> - Fixed.
>
>
> 2. Some of the listed functions are not necessary since their pairing APIs
> > are being deprecated in 1.0 already:
> >  KGroupedStream groupBy(final RichKeyValueMapper > super V, KR> selector,
> >final Serde keySerde,
> >final Serde valSerde);
> >  KStream leftJoin(final KTable table,
> >  final RichValueJoiner > V,
> > ? super VT, ? extends VR> joiner,
> >  final Serde keySerde,
> >  final Serde valSerde);
>
>
> -Fixed
>
> 3. For a few functions where we are adding three APIs for a combo of both
> > mapper / joiner, or both initializer / aggregator, or adder / subtractor,
> > I'm wondering if we can just keep one that use "rich" functions for both;
> > so that we can have less overloads and let users who only want to access
> > one of them to just use dummy parameter declarations. For example:
> >
> >  KStream join(final GlobalKTable globalKTable,
> >  final RichKeyValueMapper > super
> >  V, ? extends GK> keyValueMapper,
> >  final RichValueJoiner > V,
> > ? super GV, ? extends RV> joiner);
>
>
>
> -Agreed. Fixed.
>
>
> 4. For TimeWindowedKStream, I'm wondering why we do not make its
> > Initializer also "rich" functions? I.e.
>
>
> - It was a typo. Fixed.
>
>
> 5. We need to move "RecordContext" from o.a.k.processor.internals to
> > o.a.k.processor.
> >
> > 6. I'm not clear why we want to move `commit()` from ProcessorContext to
> > RecordContext?
> >
>
> -
> Because it makes sense logically and  to reduce code maintenance (both
> interfaces have offset() timestamp() topic() partition() methods),  I
> inherit ProcessorContext from RecordContext.
> Since we need commit() method both in ProcessorContext and in RecordContext
> I move commit() method to parent class (RecordContext).
>
>
> Cheers,
> Jeyhun
>
>
>
> On Wed, Oct 11, 2017 at 12:59 AM, Guozhang Wang 
> wrote:
>
> > Jeyhun,
> >
> > Thanks for the updated KIP, here are my comments.
> >
> > 0. RichInitializer definition seems missing.
> >
> > 1. I'd suggest moving the key parameter in the RichValueXX and
> RichReducer
> > after the value parameters, as well as in the templates; e.g.
> >
> > public interface RichValueJoiner {
> > VR apply(final V1 value1, final V2 value2, final K key, final
> > RecordContext
> > recordContext);
> > }
> >
> > My motivation is that for lambda expression in J8, users that would not
> > care about the key but only the context, or vice versa, is likely to
> write
> > it as (value1, value2, dummy, context) -> ... than putting the dummy at
> the
> > beginning of the parameter list. Generally speaking we'd like to make all
> > the "necessary" parameters prior to optional ones.
> >
> >
> > 2. Some of the listed functions are not necessary since their pairing
> APIs
> > are being deprecated in 1.0 already:
> >
> >  KGroupedStream groupBy(final RichKeyValueMapper > super V, KR> selector,
> >final Serde keySerde,
> >final Serde valSerde);
> >
> >  KStream leftJoin(final KTable table,
> >  final RichValueJoiner > V,
> > ? super VT, ? extends VR> joiner,
> >  final Serde keySerde,
> >  final Serde valSerde);
> >
> >
> >
> > 3. For a few functions where we are adding three APIs for a combo of both
> > mapper / joiner, or both initializer / aggregator, or adder / subtractor,
> > I'm wondering if we can just keep one that use "rich" functions for both;
> > so that we can have less overloads and let users who only want to access
> > one of them to just use dummy parameter declarations. For example:
> >
> >
> >  KStream join(final GlobalKTable globalKTable,
> >  final 

[GitHub] kafka pull request #4074: MINOR: add equals to SessionWindows

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4074


---

Re: [VOTE] 1.0.0 RC1

[Ted Yu]

After specifying the location of zookeeper jar:

export CLASSPATH=

The command can be executed successfully:

bin/zookeeper-server-start.sh config/zookeeper.properties

This doesn't seem to be Java 9 specific issue.

On Mon, Oct 16, 2017 at 5:30 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Guozhang,
>
> I'm not sure if this should be covered by "Java 9 support" in the RC note,
> but when I try to build jars from source using Java 9 (./gradlew
> -PscalaVersion=2.12 jar) even though the build reports as succeeded, it
> doesn't seem to have been successful:
>
> $ bin/zookeeper-server-start.sh config/zookeeper.properties
> Error: Could not find or load main class
> org.apache.zookeeper.server.quorum.QuorumPeerMain
> Caused by: java.lang.ClassNotFoundException:
> org.apache.zookeeper.server.quorum.QuorumPeerMain
>
> Please advise if I'm missing something.
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Guozhang Wang 
> To: "dev@kafka.apache.org" ,
> "us...@kafka.apache.org" , kafka-clients
> 
> Date:   10/13/2017 01:12 PM
> Subject:[VOTE] 1.0.0 RC1
>
>
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 1.0.0.
>
> It's worth noting that starting in this version we are using a different
> version protocol with three digits: *major.minor.bug-fix*
>
> Any and all testing is welcome, but the following areas are worth
> highlighting:
>
> 1. Client developers should verify that their clients can produce/consume
> to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> 2. Performance and stress testing. Heroku and LinkedIn have helped with
> this in the past (and issues have been found and fixed).
> 3. End users can verify that their apps work correctly with the new
> release.
>
> This is a major version release of Apache Kafka. It includes 29 new KIPs.
> See the release notes and release plan
> (*https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_
> confluence_pages_viewpage.action-3FpageId-3D71764913=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
> tT9k0x5RvXtHEtLzp03BA1Y8DAgHzgCXD7UjqP7oiKE=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_pages_viewpage.action-3FpageId-
> 3D71764913=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_
> OWk2y2QfKYsXitTyAHHM=tT9k0x5RvXtHEtLzp03BA1Y8DAgHzgCXD7UjqP7oiKE=
> >*)
> for more details. A few feature highlights:
>
> * Java 9 support with significantly faster TLS and CRC32C implementations
> (KIP)
> * JBOD improvements: disk failure only disables failed disk but not the
> broker (KIP-112/KIP-113)
> * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
> KIP-188, KIP-196)
> * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
> and drop compatibility "Evolving" annotations
>
> Release notes for the 1.0.0 release:
> *https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
> 7Eguozhang_kafka-2D1.0.0-2Drc1_RELEASE-5FNOTES.html=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
> xopSUD2TETEI5y8kxHM4P-jUdUKUIiUig2xVwabgDq8=
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_RELEASE-5FNOTES.
> html=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=
> xopSUD2TETEI5y8kxHM4P-jUdUKUIiUig2xVwabgDq8=
> >*
>
>
>
> *** Please download, test and vote by Tuesday, October 13, 8pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> apache.org_KEYS=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_
> OWk2y2QfKYsXitTyAHHM=FfLcWlN8ODpZ2m1KliMfp35duIxif3FNnptY5-9JKWU=
>
>
> * Release artifacts to be voted upon (source and binary):
> *https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
> 7Eguozhang_kafka-2D1.0.0-2Drc1_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_
> OWk2y2QfKYsXitTyAHHM=bcWIqj27_tkoj-fnEzcLdP8uGXyAt6gS9KUy12WF1FE=
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_=DwIBaQ=jf_
> iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=
> VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=bcWIqj27_tkoj-
> fnEzcLdP8uGXyAt6gS9KUy12WF1FE=
> >*
>
> * Maven artifacts to be voted upon:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> repository.apache.org_content_groups_staging_org_apache_
> kafka_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> 

[GitHub] kafka pull request #4078: MINOR: update exception message for KIP-120

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4078


---

Re: [VOTE] 1.0.0 RC1

[Vahid S Hashemian]

Hi Guozhang,

I'm not sure if this should be covered by "Java 9 support" in the RC note, 
but when I try to build jars from source using Java 9 (./gradlew 
-PscalaVersion=2.12 jar) even though the build reports as succeeded, it 
doesn't seem to have been successful:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
Error: Could not find or load main class 
org.apache.zookeeper.server.quorum.QuorumPeerMain
Caused by: java.lang.ClassNotFoundException: 
org.apache.zookeeper.server.quorum.QuorumPeerMain

Please advise if I'm missing something.

Thanks.
--Vahid




From:   Guozhang Wang 
To: "dev@kafka.apache.org" , 
"us...@kafka.apache.org" , kafka-clients 

Date:   10/13/2017 01:12 PM
Subject:[VOTE] 1.0.0 RC1



Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 1.0.0.

It's worth noting that starting in this version we are using a different
version protocol with three digits: *major.minor.bug-fix*

Any and all testing is welcome, but the following areas are worth
highlighting:

1. Client developers should verify that their clients can produce/consume
to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
2. Performance and stress testing. Heroku and LinkedIn have helped with
this in the past (and issues have been found and fixed).
3. End users can verify that their apps work correctly with the new 
release.

This is a major version release of Apache Kafka. It includes 29 new KIPs.
See the release notes and release plan
(*https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D71764913=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=tT9k0x5RvXtHEtLzp03BA1Y8DAgHzgCXD7UjqP7oiKE=
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D71764913=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=tT9k0x5RvXtHEtLzp03BA1Y8DAgHzgCXD7UjqP7oiKE=
>*)
for more details. A few feature highlights:

* Java 9 support with significantly faster TLS and CRC32C implementations
(KIP)
* JBOD improvements: disk failure only disables failed disk but not the
broker (KIP-112/KIP-113)
* Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
KIP-188, KIP-196)
* Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
and drop compatibility "Evolving" annotations

Release notes for the 1.0.0 release:
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_RELEASE-5FNOTES.html=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=xopSUD2TETEI5y8kxHM4P-jUdUKUIiUig2xVwabgDq8=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_RELEASE-5FNOTES.html=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=xopSUD2TETEI5y8kxHM4P-jUdUKUIiUig2xVwabgDq8=
>*



*** Please download, test and vote by Tuesday, October 13, 8pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_KEYS=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=FfLcWlN8ODpZ2m1KliMfp35duIxif3FNnptY5-9JKWU=


* Release artifacts to be voted upon (source and binary):
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=bcWIqj27_tkoj-fnEzcLdP8uGXyAt6gS9KUy12WF1FE=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=bcWIqj27_tkoj-fnEzcLdP8uGXyAt6gS9KUy12WF1FE=
>*

* Maven artifacts to be voted upon:
https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_groups_staging_org_apache_kafka_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=dKi_J-8X1TkZ83fa3hLkcO0qGcuYQ0lTxtK4o6ms5m0=


* Javadoc:
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc1_javadoc_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=VyLkHrCpgoKOD8nDthZgGw_OWk2y2QfKYsXitTyAHHM=Cz7EusxOgrGNnBmtjdqZDqPGTL3937oedTa8xm7L-9c=
<

[GitHub] kafka pull request #4078: MINOR: update exception message for KIP-120

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/4078

MINOR: update exception message for KIP-120



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka hotfix-streams

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4078


commit d0473245c452b44eb71c81bf2880ea5a4e5de099
Author: Matthias J. Sax 
Date:   2017-10-17T00:07:41Z

MINOR: update exception message for KIP-120




---

Re: [kafka-clients] Re: [VOTE] 1.0.0 RC1

[Guozhang Wang]

Thanks Jun,

I will roll out a new RC for this PR then.


Guozhang

On Mon, Oct 16, 2017 at 2:19 PM, Jun Rao  wrote:

> Hi, Guozhang,
>
> Onur found an existing performance bug in the controller when there are
> lots of partitions. The fix is simple (https://github.com/apache/
> kafka/pull/4075) and reduces the controlled shutdown time from 6.5 mins
> to 30 secs, with
> 25K partitions, RF=2 and 5 brokers.
>
> It would be useful to include this fix in 1.0.0.
>
> Thanks,
>
> Jun
>
>
> On Mon, Oct 16, 2017 at 9:55 AM, Guozhang Wang  wrote:
>
>> Hi Tom,
>>
>> Thanks for pointing it out. I meant to say Oct. 17th, Tuesday, for a 72
>> hours period.
>>
>> That being said, we need to have a lazy majority to accept a release RC
>> according to our bylaws (https://cwiki.apache.org/conf
>> luence/display/KAFKA/Bylaws). And if we cannot achieve that via thorough
>> testing within the period we will automatically extend the voting process.
>>
>>
>> Guozhang
>>
>>
>>
>> On Mon, Oct 16, 2017 at 5:09 AM, Thomas Crayford <
>> tcrayf...@salesforce.com> wrote:
>>
>>> Hi Guozhang,
>>>
>>> This says the due date on the testing is October 13th, which was the day
>>> this email was sent. Is that accurate, or is it meant to read October
>>> 17th,
>>> which is next Tuesday?
>>>
>>> I feel like this short a testing window for a 1.0 RC is a little low, as
>>> 1.0 is clearly a big announcement of stability, and folk should be given
>>> enough time to do thorough testing.
>>>
>>> Thanks
>>>
>>> Tom
>>>
>>> On Fri, Oct 13, 2017 at 9:12 PM, Guozhang Wang 
>>> wrote:
>>>
>>> > Hello Kafka users, developers and client-developers,
>>> >
>>> > This is the second candidate for release of Apache Kafka 1.0.0.
>>> >
>>> > It's worth noting that starting in this version we are using a
>>> different
>>> > version protocol with three digits: *major.minor.bug-fix*
>>> >
>>> > Any and all testing is welcome, but the following areas are worth
>>> > highlighting:
>>> >
>>> > 1. Client developers should verify that their clients can
>>> produce/consume
>>> > to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
>>> > 2. Performance and stress testing. Heroku and LinkedIn have helped with
>>> > this in the past (and issues have been found and fixed).
>>> > 3. End users can verify that their apps work correctly with the new
>>> > release.
>>> >
>>> > This is a major version release of Apache Kafka. It includes 29 new
>>> KIPs.
>>> > See the release notes and release plan
>>> > (*https://cwiki.apache.org/confluence/pages/viewpage.
>>> > action?pageId=71764913
>>> > >> ageId=71764913
>>> > >*)
>>> > for more details. A few feature highlights:
>>> >
>>> > * Java 9 support with significantly faster TLS and CRC32C
>>> implementations
>>> > (KIP)
>>> > * JBOD improvements: disk failure only disables failed disk but not the
>>> > broker (KIP-112/KIP-113)
>>> > * Newly added metrics across all the modules (KIP-164, KIP-168,
>>> KIP-187,
>>> > KIP-188, KIP-196)
>>> > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 /
>>> 161),
>>> > and drop compatibility "Evolving" annotations
>>> >
>>> > Release notes for the 1.0.0 release:
>>> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/RELEASE_NOTES.html
>>> > *
>>> >
>>> >
>>> >
>>> > *** Please download, test and vote by Tuesday, October 13, 8pm PT
>>> >
>>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>>> > http://kafka.apache.org/KEYS
>>> >
>>> > * Release artifacts to be voted upon (source and binary):
>>> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
>>> > *
>>> >
>>> > * Maven artifacts to be voted upon:
>>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> >
>>> > * Javadoc:
>>> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/javadoc/
>>> > *
>>> >
>>> > * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc1 tag:
>>> >
>>> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>>> > 9424d29dbf0a3c538215b0b98b1e6b956481e4d5
>>> >
>>> > * Documentation:
>>> > Note the documentation can't be pushed live due to changes that will
>>> not go
>>> > live until the release. You can manually verify by downloading
>>> > http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
>>> > kafka_2.11-1.0.0-site-docs.tgz
>>> >
>>> > * Successful Jenkins builds for the 1.0.0 branch:
>>> > Unit/integration tests: https://builds.apache.org/job/
>>> kafka-1.0-jdk7/31/
>>> > System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/1/
>>> >
>>> >
>>> > /**
>>> >
>>> >
>>> > Thanks,
>>> > -- Guozhang
>>> >
>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
>> --
>> You received this message because you are 

[GitHub] kafka pull request #4071: MINOR: a few web doc and javadoc fixes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4071


---

Re: Possible Feature: Topic Retention Policy

[Guozhang Wang]

Hello Alexei,

Thanks for bringing up this question. Just my 2 cents:

1. For request-response messaging, I think an alternative approach is to
use a single topic for request queue, and use one temporary topic for
response queue. I.e. everyone sends their request to a single topic, and
wait for its own topic for response. After received the response from the
other topic, they can delete the topic before leaving using the admin
client.

2. One disadvantage for having topic retention policy is that in a shared
Kafka cluster, user's expected retention policy would be quite different,
so by the end of the day we would need to have a per-topic retention
policy. Then going back to the motivation use cases, when creating this
temporary topic the user needs to specify the retention policy specifically
for that topic. I think this pattern would be similar to: client create the
topic (without specifying the retention policy), then after received
expected topic from it the client can delete the topic. Note that with the
admin client users can programmatically delete the topic after completed
using it, so it does not necessarily need to introduce administrative
headaches for operation teams.


Guozhang




On Sat, Oct 14, 2017 at 7:48 AM, Alexei Zenin  wrote:

> Hi,
>
>
> I have come across a few stack overflow posts on the subject of
> request-response type semantics through KAFKA. From some of the approaches
> that I've read developers are using KAFKA's auto topic create feature (or
> AdminClient) to dynamically create topics per request-response channel.
> They mention that the number of processes that wish to communicate can
> vary, with the processes using some form of id to create unique topics for
> themselves. See https://stackoverflow.com/questions/35535785/does-kafka-
> support-request-response-messaging and https://stackoverflow.com/
> questions/46662102/correlating-in-kafka-and-dynamic-topics/46678198 //stackoverflow.com/questions/46662102/correlating-in-kafka-
> and-dynamic-topics/46678198#46678198>.
>
>
> This approach leads to several problems however:
>
>
>   1.  The maximum number of such channels (topics) is limited by the
> memory of a ZK node (in-memory constraints since ZK is not sharded)
>   2.  ZK is best used when reads outnumber writes. Creating topics at high
> rates could affect ZK cluster performance.
>   3.  Once the communication is done by the initiating process some entity
> must delete the topic it used since it will never be used again
>
> To solve one part of this problem, I find it strange that KAFKA does not
> provide a Topic Retention Policy (not a log retention policy).
>
> This would delete topics which are considered "stale" from the KAFKA
> cluster and from ZK. By deleting topics for the user, this would reduce the
> amount of code and administrative headache stale topics currently place on
> the user.
>
> Would this be a feature that the community would find value in, while
> keeping true to KAFKA's fundamentals and not require substantial
> refactoring?
>
> Alexei
>
>
>
>
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-209 Connection String Support

[Clebert Suconic]

That works.

On Mon, Oct 16, 2017 at 6:59 PM Ted Yu  wrote:

> Can't you use IllegalArgumentException ?
>
> Some example in current code base:
>
> clients/src/main/java/org/apache/kafka/clients/Metadata.java:
>  throw new IllegalArgumentException("Max time to wait for metadata updates
> should not be < 0 milliseconds");
>
> On Mon, Oct 16, 2017 at 3:06 PM, Clebert Suconic <
> clebert.suco...@gmail.com>
> wrote:
>
> > I updated the wiki with the list on the proposed arguments.
> >
> > I also changed it to include a new Exception class that would be named
> > InvalidParameterException (since I couldn't find an existing Exception
> > class that I could reuse into this). (I could review the name or the
> > exception of course.. just my current proposal)
> >
> > On Mon, Oct 16, 2017 at 5:55 PM, Jakub Scholz  wrote:
> > > Hi Clebert,
> > >
> > > I think it would be good if this could cover not only KafkaConsumer and
> > > KafkaProducer but also the AdminClient. So that all three can be
> > configured
> > > the same way.
> > >
> > > The bootstrap servers are a list - you can provide multiple bootstrap
> > > servers. Maybe you add an example of how that will be configured. I
> > assume
> > > it will be
> > > "host:port,host2:port2;parameterName=value1;parameterName2=value2" but
> > it
> > > would be great to have it documented.
> > >
> > > Thanks & Regards
> > > Jakub
> > >
> > > On Mon, Oct 16, 2017 at 11:30 PM, Clebert Suconic <
> > clebert.suco...@gmail.com
> > >> wrote:
> > >
> > >> I would like to start a discussion about KIP-209
> > >> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 209+-+Connection+String+Support)
> > >>
> > >> This is an extension of my previous thread:
> > >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.
> > >> mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_
> > >> oyv...@mail.gmail.com%3e
> > >>
> > >> this could make the bootstrap of a consumer or producer similar to
> > >> what users are already used when connecting into other systems, being
> > >> a simple addition to Producer and Consumer, without breaking any
> > >> previous client usage.
> > >>
> > >>
> > >> --
> > >> Clebert Suconic
> > >>
> >
> >
> >
> > --
> > Clebert Suconic
> >
>
-- 
Clebert Suconic

[GitHub] kafka pull request #4077: KAFKA-5142: Added support for record headers, reus...

[rhauch]

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/4077

KAFKA-5142: Added support for record headers, reusing Kafka client's 
interfaces

*This is still a work in progress and should not be merged.*

This is a proposed PR that implements most of 
[KIP-145](https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect)
 but with some changes. The Kafka client library's `Headers` and `Header` 
interfaces are used directly so as to minimize the overhead of converting 
instances to a Connect-specific object. However, a new `ConnectHeaders` class 
is proposed to provide a fluent builder for easily constructing headers either 
in source connectors or SMTs that need to add/remove/modify headers, and a 
reader utility component for reading header values and converting to primitives.

Note that KIP-145 is still undergoing discussions, so this is provided 
merely as one possible approach.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5142

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4077.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4077


commit 5f31191af94e2a324a694bc99e4639d968389ff2
Author: Randall Hauch 
Date:   2017-10-16T22:59:50Z

KAFKA-5142: Added support for record headers, reusing Kafka client's 
interfaces




---

Re: [DISCUSS] KIP-209 Connection String Support

[Ted Yu]

Can't you use IllegalArgumentException ?

Some example in current code base:

clients/src/main/java/org/apache/kafka/clients/Metadata.java:
 throw new IllegalArgumentException("Max time to wait for metadata updates
should not be < 0 milliseconds");

On Mon, Oct 16, 2017 at 3:06 PM, Clebert Suconic 
wrote:

> I updated the wiki with the list on the proposed arguments.
>
> I also changed it to include a new Exception class that would be named
> InvalidParameterException (since I couldn't find an existing Exception
> class that I could reuse into this). (I could review the name or the
> exception of course.. just my current proposal)
>
> On Mon, Oct 16, 2017 at 5:55 PM, Jakub Scholz  wrote:
> > Hi Clebert,
> >
> > I think it would be good if this could cover not only KafkaConsumer and
> > KafkaProducer but also the AdminClient. So that all three can be
> configured
> > the same way.
> >
> > The bootstrap servers are a list - you can provide multiple bootstrap
> > servers. Maybe you add an example of how that will be configured. I
> assume
> > it will be
> > "host:port,host2:port2;parameterName=value1;parameterName2=value2" but
> it
> > would be great to have it documented.
> >
> > Thanks & Regards
> > Jakub
> >
> > On Mon, Oct 16, 2017 at 11:30 PM, Clebert Suconic <
> clebert.suco...@gmail.com
> >> wrote:
> >
> >> I would like to start a discussion about KIP-209
> >> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 209+-+Connection+String+Support)
> >>
> >> This is an extension of my previous thread:
> >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.
> >> mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_
> >> oyv...@mail.gmail.com%3e
> >>
> >> this could make the bootstrap of a consumer or producer similar to
> >> what users are already used when connecting into other systems, being
> >> a simple addition to Producer and Consumer, without breaking any
> >> previous client usage.
> >>
> >>
> >> --
> >> Clebert Suconic
> >>
>
>
>
> --
> Clebert Suconic
>

[GitHub] kafka pull request #4076: MINOR: A few javadoc fixes

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/4076

MINOR: A few javadoc fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka javadoc-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4076.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4076


commit 43817370f8673a309969477dc587545c528c06d5
Author: Jason Gustafson 
Date:   2017-10-16T22:42:39Z

MINOR: A few javadoc fixes




---

Jenkins build is back to normal : kafka-1.0-jdk7 #37

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-209 Connection String Support

[Clebert Suconic]

I updated the wiki with the list on the proposed arguments.

I also changed it to include a new Exception class that would be named
InvalidParameterException (since I couldn't find an existing Exception
class that I could reuse into this). (I could review the name or the
exception of course.. just my current proposal)

On Mon, Oct 16, 2017 at 5:55 PM, Jakub Scholz  wrote:
> Hi Clebert,
>
> I think it would be good if this could cover not only KafkaConsumer and
> KafkaProducer but also the AdminClient. So that all three can be configured
> the same way.
>
> The bootstrap servers are a list - you can provide multiple bootstrap
> servers. Maybe you add an example of how that will be configured. I assume
> it will be
> "host:port,host2:port2;parameterName=value1;parameterName2=value2" but it
> would be great to have it documented.
>
> Thanks & Regards
> Jakub
>
> On Mon, Oct 16, 2017 at 11:30 PM, Clebert Suconic > wrote:
>
>> I would like to start a discussion about KIP-209
>> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 209+-+Connection+String+Support)
>>
>> This is an extension of my previous thread:
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.
>> mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_
>> oyv...@mail.gmail.com%3e
>>
>> this could make the bootstrap of a consumer or producer similar to
>> what users are already used when connecting into other systems, being
>> a simple addition to Producer and Consumer, without breaking any
>> previous client usage.
>>
>>
>> --
>> Clebert Suconic
>>



-- 
Clebert Suconic

Re: [DISCUSS] KIP-209 Connection String Support

[Jakub Scholz]

Hi Clebert,

I think it would be good if this could cover not only KafkaConsumer and
KafkaProducer but also the AdminClient. So that all three can be configured
the same way.

The bootstrap servers are a list - you can provide multiple bootstrap
servers. Maybe you add an example of how that will be configured. I assume
it will be
"host:port,host2:port2;parameterName=value1;parameterName2=value2" but it
would be great to have it documented.

Thanks & Regards
Jakub

On Mon, Oct 16, 2017 at 11:30 PM, Clebert Suconic  wrote:

> I would like to start a discussion about KIP-209
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 209+-+Connection+String+Support)
>
> This is an extension of my previous thread:
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.
> mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_
> oyv...@mail.gmail.com%3e
>
> this could make the bootstrap of a consumer or producer similar to
> what users are already used when connecting into other systems, being
> a simple addition to Producer and Consumer, without breaking any
> previous client usage.
>
>
> --
> Clebert Suconic
>

Re: [DISCUSS] KIP-209 Connection String Support

[Ted Yu]

bq. I was waiting my email to go through the servers

http://search-hadoop.com/ indexes mailing lists actively. The delay is very
short.

FYI

On Mon, Oct 16, 2017 at 2:44 PM, Clebert Suconic 
wrote:

> On Mon, Oct 16, 2017 at 5:41 PM, Ted Yu  wrote:
> > Please update link for Discussion thread and JIRA
>
> sure thing... I was waiting my email to go through the servers... so I
> could get the link.
>
>
> >
> > There're two TBD's for Invalid conversion and parameters. Can you fill
> them
> > out ?
>
> will do.. I was hoping to get feedback from this discussion on what
> could be the best exception class to be thrown though.. will do some
> research.
>

Re: [DISCUSS] KIP-209 Connection String Support

[Clebert Suconic]

On Mon, Oct 16, 2017 at 5:41 PM, Ted Yu  wrote:
> Please update link for Discussion thread and JIRA

sure thing... I was waiting my email to go through the servers... so I
could get the link.


>
> There're two TBD's for Invalid conversion and parameters. Can you fill them
> out ?

will do.. I was hoping to get feedback from this discussion on what
could be the best exception class to be thrown though.. will do some
research.

Re: [DISCUSS] KIP-209 Connection String Support

[Ted Yu]

Please update link for Discussion thread and JIRA

There're two TBD's for Invalid conversion and parameters. Can you fill them
out ?

Thanks

On Mon, Oct 16, 2017 at 2:30 PM, Clebert Suconic 
wrote:

> I would like to start a discussion about KIP-209
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 209+-+Connection+String+Support)
>
> This is an extension of my previous thread:
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.
> mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=S8M4KyJxAt_
> oyv...@mail.gmail.com%3e
>
> this could make the bootstrap of a consumer or producer similar to
> what users are already used when connecting into other systems, being
> a simple addition to Producer and Consumer, without breaking any
> previous client usage.
>
>
> --
> Clebert Suconic
>

[DISCUSS] KIP-209 Connection String Support

[Clebert Suconic]

I would like to start a discussion about KIP-209
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-209+-+Connection+String+Support)

This is an extension of my previous thread:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201710.mbox/%3cCAKF+bsoFbN13D-u20tUsP6G+aHX4BUNk=s8m4kyjxat_oyv...@mail.gmail.com%3e

this could make the bootstrap of a consumer or producer similar to
what users are already used when connecting into other systems, being
a simple addition to Producer and Consumer, without breaking any
previous client usage.


-- 
Clebert Suconic

[GitHub] kafka pull request #4075: MINOR: reduce partition state machine debug loggin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4075


---

Re: [kafka-clients] Re: [VOTE] 1.0.0 RC1

[Jun Rao]

Hi, Guozhang,

Onur found an existing performance bug in the controller when there are
lots of partitions. The fix is simple (
https://github.com/apache/kafka/pull/4075) and reduces the controlled
shutdown time from 6.5 mins to 30 secs, with
25K partitions, RF=2 and 5 brokers.

It would be useful to include this fix in 1.0.0.

Thanks,

Jun


On Mon, Oct 16, 2017 at 9:55 AM, Guozhang Wang  wrote:

> Hi Tom,
>
> Thanks for pointing it out. I meant to say Oct. 17th, Tuesday, for a 72
> hours period.
>
> That being said, we need to have a lazy majority to accept a release RC
> according to our bylaws (https://cwiki.apache.org/
> confluence/display/KAFKA/Bylaws). And if we cannot achieve that via
> thorough testing within the period we will automatically extend the voting
> process.
>
>
> Guozhang
>
>
>
> On Mon, Oct 16, 2017 at 5:09 AM, Thomas Crayford  > wrote:
>
>> Hi Guozhang,
>>
>> This says the due date on the testing is October 13th, which was the day
>> this email was sent. Is that accurate, or is it meant to read October
>> 17th,
>> which is next Tuesday?
>>
>> I feel like this short a testing window for a 1.0 RC is a little low, as
>> 1.0 is clearly a big announcement of stability, and folk should be given
>> enough time to do thorough testing.
>>
>> Thanks
>>
>> Tom
>>
>> On Fri, Oct 13, 2017 at 9:12 PM, Guozhang Wang 
>> wrote:
>>
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the second candidate for release of Apache Kafka 1.0.0.
>> >
>> > It's worth noting that starting in this version we are using a different
>> > version protocol with three digits: *major.minor.bug-fix*
>> >
>> > Any and all testing is welcome, but the following areas are worth
>> > highlighting:
>> >
>> > 1. Client developers should verify that their clients can
>> produce/consume
>> > to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
>> > 2. Performance and stress testing. Heroku and LinkedIn have helped with
>> > this in the past (and issues have been found and fixed).
>> > 3. End users can verify that their apps work correctly with the new
>> > release.
>> >
>> > This is a major version release of Apache Kafka. It includes 29 new
>> KIPs.
>> > See the release notes and release plan
>> > (*https://cwiki.apache.org/confluence/pages/viewpage.
>> > action?pageId=71764913
>> > > pageId=71764913
>> > >*)
>> > for more details. A few feature highlights:
>> >
>> > * Java 9 support with significantly faster TLS and CRC32C
>> implementations
>> > (KIP)
>> > * JBOD improvements: disk failure only disables failed disk but not the
>> > broker (KIP-112/KIP-113)
>> > * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
>> > KIP-188, KIP-196)
>> > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 /
>> 161),
>> > and drop compatibility "Evolving" annotations
>> >
>> > Release notes for the 1.0.0 release:
>> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/RELEASE_NOTES.html
>> > *
>> >
>> >
>> >
>> > *** Please download, test and vote by Tuesday, October 13, 8pm PT
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > http://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
>> > *
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> >
>> > * Javadoc:
>> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/javadoc/
>> > *
>> >
>> > * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc1 tag:
>> >
>> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> > 9424d29dbf0a3c538215b0b98b1e6b956481e4d5
>> >
>> > * Documentation:
>> > Note the documentation can't be pushed live due to changes that will
>> not go
>> > live until the release. You can manually verify by downloading
>> > http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
>> > kafka_2.11-1.0.0-site-docs.tgz
>> >
>> > * Successful Jenkins builds for the 1.0.0 branch:
>> > Unit/integration tests: https://builds.apache.org/job/
>> kafka-1.0-jdk7/31/
>> > System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/1/
>> >
>> >
>> > /**
>> >
>> >
>> > Thanks,
>> > -- Guozhang
>> >
>>
>
>
>
> --
> -- Guozhang
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at 

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

[Bill Bejeck]

+1

Thanks,
Bill

On Fri, Oct 13, 2017 at 6:36 PM, Ted Yu  wrote:

> +1
>
> On Fri, Oct 13, 2017 at 3:32 PM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> >
> >
> > On 9/11/17 3:04 PM, Jorge Esteban Quilcate Otoya wrote:
> > > Hi All,
> > >
> > > It seems that there is no further concern with the KIP-171.
> > > At this point we would like to start the voting process.
> > >
> > > The KIP can be found here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application
> > >
> > >
> > > Thanks!
> > >
> >
> >
>

[jira] [Created] (KAFKA-6064) Cluster hung when the controller tried to delete a bunch of topics

[Chaitanya GSK (JIRA)]

Chaitanya GSK created KAFKA-6064:


 Summary: Cluster hung when the controller tried to delete a bunch 
of topics 
 Key: KAFKA-6064
 URL: https://issues.apache.org/jira/browse/KAFKA-6064
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 0.8.2.1
 Environment: rhel 6, 12 core, 48GB 
Reporter: Chaitanya GSK


Hi, 

We have been using 0.8.2.1 in our kafka cluster and we had a full cluster 
outage when we programmatically tried to delete 220 topics and the controller 
got hung and went out of memory. This has somehow led to the whole cluster 
outage and the clients were not able to push the data at the right rate. AFAIK, 
controller shouldn't impact the write rate to the fellow brokers and in this 
case, it did. Below is the client error.

[WARN] Failed to send kafka.producer.async request with correlation id 
1613935688 to broker 44 with data for partitions 
[topic_2,65],[topic_2,167],[topic_3,2],[topic_4,0],[topic_5,30],[topic_2,48],[topic_2,150]
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[?:1.8.0_60]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) 
~[?:1.8.0_60]
at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[?:1.8.0_60]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504) 
~[?:1.8.0_60]
at java.nio.channels.SocketChannel.write(SocketChannel.java:502) 
~[?:1.8.0_60]
at 
kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56) 
~[stormjar.jar:?]
at kafka.network.Send$class.writeCompletely(Transmission.scala:75) 
~[stormjar.jar:?]
at 
kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
 ~[stormjar.jar:?]
at kafka.network.BlockingChannel.send(BlockingChannel.scala:103) 
~[stormjar.jar:?]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) 
~[stormjar.jar:?]
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
 ~[stormjar.jar:?]
at 
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:103)
 ~[stormjar.jar:?]
at 
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
 ~[stormjar.jar:?]
at 
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
 ~[stormjar.jar:?]
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) ~[stormjar.jar:?]
at 
kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:102)
 ~[stormjar.jar:?]
at 
kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:102) 
~[stormjar.jar:?]
at 
kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:102) 
~[stormjar.jar:?]
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) ~[stormjar.jar:?]
at kafka.producer.SyncProducer.send(SyncProducer.scala:101) 
~[stormjar.jar:?]
at 
kafka.producer.async.YamasKafkaEventHandler.kafka$producer$async$YamasKafkaEventHandler$$send(YamasKafkaEventHandler.scala:481)
 [stormjar.jar:?]
at 
kafka.producer.async.YamasKafkaEventHandler$$anonfun$dispatchSerializedData$2.apply(YamasKafkaEventHandler.scala:144)
 [stormjar.jar:?]
at 
kafka.producer.async.YamasKafkaEventHandler$$anonfun$dispatchSerializedData$2.apply(YamasKafkaEventHandler.scala:138)
 [stormjar.jar:?]
at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
 [stormjar.jar:?]
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
[stormjar.jar:?]
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) 
[stormjar.jar:?]
at 
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) 
[stormjar.jar:?]
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) 
[stormjar.jar:?]
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) 
[stormjar.jar:?]
at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) 
[stormjar.jar:?]
at 
kafka.producer.async.YamasKafkaEventHandler.dispatchSerializedData(YamasKafkaEventHandler.scala:138)
 [stormjar.jar:?]
at 
kafka.producer.async.YamasKafkaEventHandler.handle(YamasKafkaEventHandler.scala:79)
 [stormjar.jar:?]
at 
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
 [stormjar.jar:?]
at 
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
 [stormjar.jar:?]
at 
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
 [stormjar.jar:?]
at 

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

[Xavier Léauté]

Thank you Richard! Do you or Guozhang have any thoughts on my suggestions
to use fetchAll() and fetchAll(timeFrom, timeTo) and reserve the "range"
keyword for when we query a specific range of keys?

Xavier

On Sat, Oct 14, 2017 at 2:32 PM Richard Yu 
wrote:

> Thanks for the clarifications, Xavier.
> I have removed most of the methods except for keys() and all() which has
> been renamed to Guozhang Wang's suggestions.
>
> Hope this helps.
>
> On Fri, Oct 13, 2017 at 3:28 PM, Xavier Léauté 
> wrote:
>
> > Thanks for the KIP Richard, this is a very useful addition!
> >
> > As far as the API changes, I just have a few comments on the methods that
> > don't seem directly related to the KIP title, and naming of course :).
> > On the implementation, see my notes further down that will hopefully
> > clarify a few things.
> >
> > Regarding the "bonus" methods:
> > I agree with Guozhang that the KIP lacks proper motivation for adding the
> > min, max, and allLatest methods.
> > It is also not clear to me what min and max would really mean, what
> > ordering do we refer to here? Are we first ordering by time, then key, or
> > first by key, then time?
> > The allLatest method might be useful, but I don't really see how it would
> > be used in practice if we have to scan the entire range of keys for all
> the
> > state stores, every single time.
> >
> > Maybe we could flesh the motivation behind those extra methods, but in
> the
> > interest of time, and moving the KIP forward it might make sense to file
> a
> > follow-up once we have more concrete use-cases.
> >
> > On naming:
> > I also agree with Guozhang that "keys()" should be renamed. It feels a
> bit
> > of a misnomer, since it not only returns keys, but also the values.
> >
> > As far as what to rename it to, I would argue we already have some
> > discrepancy between key-value stores using range() vs. window stores
> using
> > fetch().
> > I assume we called the window method "fetch" instead of "get" because you
> > might get back more than one window for the requested key.
> >
> > If we wanted to make things consistent with both existing key-value store
> > naming and window store naming, we could do the following:
> > Decide that "all" always refers to the entire range of keys, independent
> of
> > the window and similarly "range" always refers to a particular range of
> > keys, irrespective of the window.
> > We can then prefix methods with "fetch" to indicate that more than one
> > window may be returned for each key in the range.
> >
> > This would give us:
> > - a new fetchAll() method for all the keys, which makes it clear that you
> > might get back the same key in different windows
> > - a new fetchAll(timeFrom, timeTo) method to get all the keys in a given
> > time range, again with possibly more than one window per key
> > - and we'd have to rename fetch(K,K,long, long) to fetchRange(K, K, long,
> > long)  and deprecate the old one to indicate a range of keys
> >
> > One inconsistency I noted: the "Proposed Changes" section in your KIP
> talks
> > about a "range(timeFrom, timeTo)" method, I think you meant to refer to
> the
> > all(from, to) method, but I'm sure you'll fix that once we decide on
> > naming.
> >
> > On the implementation side:
> > You mentioned that caching and rocksdb store have very different
> key/value
> > structures, and while it appears to be that way on the surface, the
> > structure between the two is actually very similar. Keys in the cache are
> > prefixed with a segment ID to ensure the ordering in the cache stays
> > consistent with the rocksdb implementation, which maintains multiple
> > rocksdb instances, one for each segment. So we just "artificially" mirror
> > the segment structure in the cache.
> >
> > The reason for keeping the ordering consistent is pretty simple: keep in
> > mind that when we query a cached window store we are effectively querying
> > both the cache and the persistent rocksdb store at the same time, merging
> > results from both. To make that merge as painless as possible, we ensure
> > the ordering is consistent when querying a range of keys in both stores.
> >
> > Also keep in mind CompositeReadonlyWindowStore, which wraps multiple
> window
> > stores within a topology.
> >
> > Hope this clarifies some of the less trivial parts of caching window
> store.
> >
> > Cheers,
> > Xavier
> >
> > On Sun, Oct 8, 2017 at 9:21 PM Guozhang Wang  wrote:
> >
> > > Richard, Matthias:
> > >
> > > 0. Could you describe a bit what are the possible use cases of
> > `allLatest`,
> > > `minKey` and `maxKey`? I'd prefer keeping the APIs to add at a minimum
> > > necessary amount, to avoid a swamp of new APIs that no one would really
> > use
> > > but just complicated the internal code base.
> > >
> > > 1. One minor comment on the other two new APIs: could we rename `keys`
> to
> > > `all` and `all` to `range` to be consistent with the other 

[GitHub] kafka pull request #4075: MINOR: reduce partition state machine debug loggin...

[onurkaraman]

GitHub user onurkaraman opened a pull request:

https://github.com/apache/kafka/pull/4075

MINOR: reduce partition state machine debug logging

PartitionStateMachine.electLeaderForPartition logs all partition states in 
the cluster. This leads to quadratic logging behavior since 
PartitionStateMachine.electLeaderForPartition itself gets called on a 
per-partition basis.

This patch reduces the logging so that only the single partition undergoing 
leader election gets its state logged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/onurkaraman/kafka 
reduce-partition-state-machine-debug-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4075


commit dc6bfa553e73ffccd1e604963e076c78d8ddcd69
Author: Onur Karaman 
Date:   2017-10-16T17:36:16Z

MINOR: reduce partition state machine debug logging

PartitionStateMachine.electLeaderForPartition logs all partition states in 
the cluster. This leads to quadratic logging behavior since 
PartitionStateMachine.electLeaderForPartition itself gets called on a 
per-partition basis.

This patch reduces the logging so that only the single partition undergoing 
leader election gets its state logged.




---

Re: [VOTE] 1.0.0 RC1

[Guozhang Wang]

Hi Tom,

Thanks for pointing it out. I meant to say Oct. 17th, Tuesday, for a 72
hours period.

That being said, we need to have a lazy majority to accept a release RC
according to our bylaws (
https://cwiki.apache.org/confluence/display/KAFKA/Bylaws). And if we cannot
achieve that via thorough testing within the period we will automatically
extend the voting process.


Guozhang



On Mon, Oct 16, 2017 at 5:09 AM, Thomas Crayford 
wrote:

> Hi Guozhang,
>
> This says the due date on the testing is October 13th, which was the day
> this email was sent. Is that accurate, or is it meant to read October 17th,
> which is next Tuesday?
>
> I feel like this short a testing window for a 1.0 RC is a little low, as
> 1.0 is clearly a big announcement of stability, and folk should be given
> enough time to do thorough testing.
>
> Thanks
>
> Tom
>
> On Fri, Oct 13, 2017 at 9:12 PM, Guozhang Wang  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 1.0.0.
> >
> > It's worth noting that starting in this version we are using a different
> > version protocol with three digits: *major.minor.bug-fix*
> >
> > Any and all testing is welcome, but the following areas are worth
> > highlighting:
> >
> > 1. Client developers should verify that their clients can produce/consume
> > to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> > 2. Performance and stress testing. Heroku and LinkedIn have helped with
> > this in the past (and issues have been found and fixed).
> > 3. End users can verify that their apps work correctly with the new
> > release.
> >
> > This is a major version release of Apache Kafka. It includes 29 new KIPs.
> > See the release notes and release plan
> > (*https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=71764913
> >  action?pageId=71764913
> > >*)
> > for more details. A few feature highlights:
> >
> > * Java 9 support with significantly faster TLS and CRC32C implementations
> > (KIP)
> > * JBOD improvements: disk failure only disables failed disk but not the
> > broker (KIP-112/KIP-113)
> > * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
> > KIP-188, KIP-196)
> > * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
> > and drop compatibility "Evolving" annotations
> >
> > Release notes for the 1.0.0 release:
> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/RELEASE_NOTES.html
> > *
> >
> >
> >
> > *** Please download, test and vote by Tuesday, October 13, 8pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
> > *
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/javadoc/
> > *
> >
> > * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc1 tag:
> >
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> > 9424d29dbf0a3c538215b0b98b1e6b956481e4d5
> >
> > * Documentation:
> > Note the documentation can't be pushed live due to changes that will not
> go
> > live until the release. You can manually verify by downloading
> > http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
> > kafka_2.11-1.0.0-site-docs.tgz
> >
> > * Successful Jenkins builds for the 1.0.0 branch:
> > Unit/integration tests: https://builds.apache.org/job/kafka-1.0-jdk7/31/
> > System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/1/
> >
> >
> > /**
> >
> >
> > Thanks,
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

[Randall Hauch]

The broker's configuration options are "listeners" (plural) and
"listeners.security.protocol.map". I agree that following the pattern set
by the broker is better, so these are really good ideas. However, at this
point I don't see a need for the "listeners.security.procotol.map", which
for the broker must be set if the listener name is not a security protocol.
Can we not simply just allow "HTTP" and "HTTPS" as the names of the
listeners (rather than the broker's "PLAINTEXT", "SSL", etc.)? If so, then
for example "listeners" might be set to "http://myhost:8081,
https://myhost:80;, which seems to work out nicely without needing listener
names other than security protocols.

I also like using the worker's SSL and SASL security configs by default if
"https" is included in the listener, but allowing the overriding of this
via other additional properties. Here, I'm not a big fan of
"listeners.name.https.*" prefix, which I think is pretty verbose, but I
could see "listener.https.*" as a prefix. This allows us to add other
security protocols at some point, if that ever becomes necessary.

+1 for continuing down this road. Nice work.

On Mon, Oct 16, 2017 at 9:51 AM, Ted Yu  wrote:

> +1 to this proposal.
>
> On Mon, Oct 16, 2017 at 7:49 AM, Jakub Scholz  wrote:
>
> > I was having some more thoughts about it. We can simply take over what
> > Kafka broker implements for the listeners:
> > - We can take over the "listener" and "listener.security.protocol.map"
> > options to define multiple REST listeners and the security protocol they
> > should use
> > - The HTTPS interface will by default use the default configuration
> options
> > ("ssl.keystore.localtion" etc.). But if desired, the values can be
> > overridden for given listener (again, as in Kafka broker "listener.name
> > ..ssl.keystore.location")
> >
> > This should address both issues raised. But before I incorporate it into
> > the KIP, I would love to get some feedback if this sounds OK. Please let
> me
> > know what do you think ...
> >
> > Jakub
> >
> > On Sun, Oct 15, 2017 at 12:23 AM, Jakub Scholz  wrote:
> >
> > > I agree, adding both HTTP and HTTPS is not complicated. I just didn't
> saw
> > > the use case for it. But I can add it. Would you add just support for a
> > > single HTTP and single HTTPS interface? Or do you see some value even
> in
> > > allowing more than 2 interfaces (for example one HTTP and two HTTPS
> with
> > > different configuration)? It could be done similarly to how the Kafka
> > > broker does it through the "listener" configuration parameter with
> comma
> > > separated list. What do you think?
> > >
> > > As for the "rest" prefix - if we remove it, some of the same
> > configuration
> > > options are already used today as the option for connecting from Kafka
> > > Connect to Kafka broker. So I'm not sure we should mix them. I can
> > > definitely imagine some cases where the client SSL configuration will
> not
> > > be the same as the REST HTTPS configuration. That is why I added the
> > > prefix. If we remove the prefix, how would you deal with this?
> > >
> > > On Fri, Oct 13, 2017 at 6:25 PM, Randall Hauch 
> wrote:
> > >
> > >> Also, do we need these properties to be preceded with `rest`? I'd
> argue
> > >> that we're just configuring the worker's SSL information, and that the
> > >> REST
> > >> API would just use that. If we added another non-REST API, we'd want
> to
> > >> use
> > >> the same security configuration.
> > >>
> > >> It's not that complicated in Jetty to support both "http" and "https"
> > >> simultaneously, so IMO we should add that from the beginning.
> > >>
> > >> On Fri, Oct 13, 2017 at 9:34 AM, Randall Hauch 
> > wrote:
> > >>
> > >> > It'd be useful to specify the default values for the configuration
> > >> > properties.
> > >> >
> > >> > On Tue, Oct 10, 2017 at 2:53 AM, Jakub Scholz 
> > wrote:
> > >> >
> > >> >> FYI: Based on Ewen's suggestion from the related JIRA, I added a
> > >> >> clarification to the KIP that it doesn't do anything around
> > >> authorization
> > >> >> /
> > >> >> ACLs. While authorization / ACLs would be for sure valuable
> feature I
> > >> >> would
> > >> >> prefer to leave it for different KIP.
> > >> >>
> > >> >> Jakub
> > >> >>
> > >> >> On Mon, Oct 9, 2017 at 5:25 PM, Jakub Scholz 
> > wrote:
> > >> >>
> > >> >> > Hi,
> > >> >> >
> > >> >> > I would like to start a discussion about KIP-208: Add SSL support
> > to
> > >> >> Kafka
> > >> >> > Connect REST interface (https://cwiki.apache.org/
> > >> >> > confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+
> > >> >> > Kafka+Connect+REST+interface).
> > >> >> >
> > >> >> > I think this would be useful feature to improve the security of
> > Kafka
> > >> >> > Connect.
> > >> >> >
> > >> >> > Thanks & Regards
> > >> >> > Jakub
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Jeyhun Karimov]

Hi,

Thanks for the feedback.


0. RichInitializer definition seems missing.



- Fixed.


 I'd suggest moving the key parameter in the RichValueXX and RichReducer
> after the value parameters, as well as in the templates; e.g.
> public interface RichValueJoiner {
> VR apply(final V1 value1, final V2 value2, final K key, final
> RecordContext
> recordContext);
> }



- Fixed.


2. Some of the listed functions are not necessary since their pairing APIs
> are being deprecated in 1.0 already:
>  KGroupedStream groupBy(final RichKeyValueMapper super V, KR> selector,
>final Serde keySerde,
>final Serde valSerde);
>  KStream leftJoin(final KTable table,
>  final RichValueJoiner V,
> ? super VT, ? extends VR> joiner,
>  final Serde keySerde,
>  final Serde valSerde);


-Fixed

3. For a few functions where we are adding three APIs for a combo of both
> mapper / joiner, or both initializer / aggregator, or adder / subtractor,
> I'm wondering if we can just keep one that use "rich" functions for both;
> so that we can have less overloads and let users who only want to access
> one of them to just use dummy parameter declarations. For example:
>
>  KStream join(final GlobalKTable globalKTable,
>  final RichKeyValueMapper super
>  V, ? extends GK> keyValueMapper,
>  final RichValueJoiner V,
> ? super GV, ? extends RV> joiner);



-Agreed. Fixed.


4. For TimeWindowedKStream, I'm wondering why we do not make its
> Initializer also "rich" functions? I.e.


- It was a typo. Fixed.


5. We need to move "RecordContext" from o.a.k.processor.internals to
> o.a.k.processor.
>
> 6. I'm not clear why we want to move `commit()` from ProcessorContext to
> RecordContext?
>

-
Because it makes sense logically and  to reduce code maintenance (both
interfaces have offset() timestamp() topic() partition() methods),  I
inherit ProcessorContext from RecordContext.
Since we need commit() method both in ProcessorContext and in RecordContext
I move commit() method to parent class (RecordContext).


Cheers,
Jeyhun



On Wed, Oct 11, 2017 at 12:59 AM, Guozhang Wang  wrote:

> Jeyhun,
>
> Thanks for the updated KIP, here are my comments.
>
> 0. RichInitializer definition seems missing.
>
> 1. I'd suggest moving the key parameter in the RichValueXX and RichReducer
> after the value parameters, as well as in the templates; e.g.
>
> public interface RichValueJoiner {
> VR apply(final V1 value1, final V2 value2, final K key, final
> RecordContext
> recordContext);
> }
>
> My motivation is that for lambda expression in J8, users that would not
> care about the key but only the context, or vice versa, is likely to write
> it as (value1, value2, dummy, context) -> ... than putting the dummy at the
> beginning of the parameter list. Generally speaking we'd like to make all
> the "necessary" parameters prior to optional ones.
>
>
> 2. Some of the listed functions are not necessary since their pairing APIs
> are being deprecated in 1.0 already:
>
>  KGroupedStream groupBy(final RichKeyValueMapper super V, KR> selector,
>final Serde keySerde,
>final Serde valSerde);
>
>  KStream leftJoin(final KTable table,
>  final RichValueJoiner V,
> ? super VT, ? extends VR> joiner,
>  final Serde keySerde,
>  final Serde valSerde);
>
>
>
> 3. For a few functions where we are adding three APIs for a combo of both
> mapper / joiner, or both initializer / aggregator, or adder / subtractor,
> I'm wondering if we can just keep one that use "rich" functions for both;
> so that we can have less overloads and let users who only want to access
> one of them to just use dummy parameter declarations. For example:
>
>
>  KStream join(final GlobalKTable globalKTable,
>  final RichKeyValueMapper super
>  V, ? extends GK> keyValueMapper,
>  final RichValueJoiner V,
> ? super GV, ? extends RV> joiner);
>
>  KTable aggregate(final RichInitializer initializer,
>  final RichAggregator
> aggregator,
>  final Materialized byte[]>> materialized);
>
> Similarly for KGroupedTable, a bunch of aggregate() are deprecated so we do
> not need to add its rich functions any more.
>
>
> 4. For TimeWindowedKStream, I'm wondering why we do not make its
> Initializer also "rich" functions? I.e.
>
>  KTable aggregate(final RichInitializer
> initializer,
> 

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

[Ted Yu]

+1 to this proposal.

On Mon, Oct 16, 2017 at 7:49 AM, Jakub Scholz  wrote:

> I was having some more thoughts about it. We can simply take over what
> Kafka broker implements for the listeners:
> - We can take over the "listener" and "listener.security.protocol.map"
> options to define multiple REST listeners and the security protocol they
> should use
> - The HTTPS interface will by default use the default configuration options
> ("ssl.keystore.localtion" etc.). But if desired, the values can be
> overridden for given listener (again, as in Kafka broker "listener.name
> ..ssl.keystore.location")
>
> This should address both issues raised. But before I incorporate it into
> the KIP, I would love to get some feedback if this sounds OK. Please let me
> know what do you think ...
>
> Jakub
>
> On Sun, Oct 15, 2017 at 12:23 AM, Jakub Scholz  wrote:
>
> > I agree, adding both HTTP and HTTPS is not complicated. I just didn't saw
> > the use case for it. But I can add it. Would you add just support for a
> > single HTTP and single HTTPS interface? Or do you see some value even in
> > allowing more than 2 interfaces (for example one HTTP and two HTTPS with
> > different configuration)? It could be done similarly to how the Kafka
> > broker does it through the "listener" configuration parameter with comma
> > separated list. What do you think?
> >
> > As for the "rest" prefix - if we remove it, some of the same
> configuration
> > options are already used today as the option for connecting from Kafka
> > Connect to Kafka broker. So I'm not sure we should mix them. I can
> > definitely imagine some cases where the client SSL configuration will not
> > be the same as the REST HTTPS configuration. That is why I added the
> > prefix. If we remove the prefix, how would you deal with this?
> >
> > On Fri, Oct 13, 2017 at 6:25 PM, Randall Hauch  wrote:
> >
> >> Also, do we need these properties to be preceded with `rest`? I'd argue
> >> that we're just configuring the worker's SSL information, and that the
> >> REST
> >> API would just use that. If we added another non-REST API, we'd want to
> >> use
> >> the same security configuration.
> >>
> >> It's not that complicated in Jetty to support both "http" and "https"
> >> simultaneously, so IMO we should add that from the beginning.
> >>
> >> On Fri, Oct 13, 2017 at 9:34 AM, Randall Hauch 
> wrote:
> >>
> >> > It'd be useful to specify the default values for the configuration
> >> > properties.
> >> >
> >> > On Tue, Oct 10, 2017 at 2:53 AM, Jakub Scholz 
> wrote:
> >> >
> >> >> FYI: Based on Ewen's suggestion from the related JIRA, I added a
> >> >> clarification to the KIP that it doesn't do anything around
> >> authorization
> >> >> /
> >> >> ACLs. While authorization / ACLs would be for sure valuable feature I
> >> >> would
> >> >> prefer to leave it for different KIP.
> >> >>
> >> >> Jakub
> >> >>
> >> >> On Mon, Oct 9, 2017 at 5:25 PM, Jakub Scholz 
> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I would like to start a discussion about KIP-208: Add SSL support
> to
> >> >> Kafka
> >> >> > Connect REST interface (https://cwiki.apache.org/
> >> >> > confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+
> >> >> > Kafka+Connect+REST+interface).
> >> >> >
> >> >> > I think this would be useful feature to improve the security of
> Kafka
> >> >> > Connect.
> >> >> >
> >> >> > Thanks & Regards
> >> >> > Jakub
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

[Jakub Scholz]

I was having some more thoughts about it. We can simply take over what
Kafka broker implements for the listeners:
- We can take over the "listener" and "listener.security.protocol.map"
options to define multiple REST listeners and the security protocol they
should use
- The HTTPS interface will by default use the default configuration options
("ssl.keystore.localtion" etc.). But if desired, the values can be
overridden for given listener (again, as in Kafka broker "listener.name
..ssl.keystore.location")

This should address both issues raised. But before I incorporate it into
the KIP, I would love to get some feedback if this sounds OK. Please let me
know what do you think ...

Jakub

On Sun, Oct 15, 2017 at 12:23 AM, Jakub Scholz  wrote:

> I agree, adding both HTTP and HTTPS is not complicated. I just didn't saw
> the use case for it. But I can add it. Would you add just support for a
> single HTTP and single HTTPS interface? Or do you see some value even in
> allowing more than 2 interfaces (for example one HTTP and two HTTPS with
> different configuration)? It could be done similarly to how the Kafka
> broker does it through the "listener" configuration parameter with comma
> separated list. What do you think?
>
> As for the "rest" prefix - if we remove it, some of the same configuration
> options are already used today as the option for connecting from Kafka
> Connect to Kafka broker. So I'm not sure we should mix them. I can
> definitely imagine some cases where the client SSL configuration will not
> be the same as the REST HTTPS configuration. That is why I added the
> prefix. If we remove the prefix, how would you deal with this?
>
> On Fri, Oct 13, 2017 at 6:25 PM, Randall Hauch  wrote:
>
>> Also, do we need these properties to be preceded with `rest`? I'd argue
>> that we're just configuring the worker's SSL information, and that the
>> REST
>> API would just use that. If we added another non-REST API, we'd want to
>> use
>> the same security configuration.
>>
>> It's not that complicated in Jetty to support both "http" and "https"
>> simultaneously, so IMO we should add that from the beginning.
>>
>> On Fri, Oct 13, 2017 at 9:34 AM, Randall Hauch  wrote:
>>
>> > It'd be useful to specify the default values for the configuration
>> > properties.
>> >
>> > On Tue, Oct 10, 2017 at 2:53 AM, Jakub Scholz  wrote:
>> >
>> >> FYI: Based on Ewen's suggestion from the related JIRA, I added a
>> >> clarification to the KIP that it doesn't do anything around
>> authorization
>> >> /
>> >> ACLs. While authorization / ACLs would be for sure valuable feature I
>> >> would
>> >> prefer to leave it for different KIP.
>> >>
>> >> Jakub
>> >>
>> >> On Mon, Oct 9, 2017 at 5:25 PM, Jakub Scholz  wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > I would like to start a discussion about KIP-208: Add SSL support to
>> >> Kafka
>> >> > Connect REST interface (https://cwiki.apache.org/
>> >> > confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+
>> >> > Kafka+Connect+REST+interface).
>> >> >
>> >> > I think this would be useful feature to improve the security of Kafka
>> >> > Connect.
>> >> >
>> >> > Thanks & Regards
>> >> > Jakub
>> >> >
>> >>
>> >
>> >
>>
>
>

[GitHub] kafka pull request #4074: MINOR: add equals to SessionWindows

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/4074

MINOR: add equals to SessionWindows



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka minor-session-window-equals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4074






---

Re: [VOTE] 1.0.0 RC1

[Thomas Crayford]

Hi Guozhang,

This says the due date on the testing is October 13th, which was the day
this email was sent. Is that accurate, or is it meant to read October 17th,
which is next Tuesday?

I feel like this short a testing window for a 1.0 RC is a little low, as
1.0 is clearly a big announcement of stability, and folk should be given
enough time to do thorough testing.

Thanks

Tom

On Fri, Oct 13, 2017 at 9:12 PM, Guozhang Wang  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 1.0.0.
>
> It's worth noting that starting in this version we are using a different
> version protocol with three digits: *major.minor.bug-fix*
>
> Any and all testing is welcome, but the following areas are worth
> highlighting:
>
> 1. Client developers should verify that their clients can produce/consume
> to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> 2. Performance and stress testing. Heroku and LinkedIn have helped with
> this in the past (and issues have been found and fixed).
> 3. End users can verify that their apps work correctly with the new
> release.
>
> This is a major version release of Apache Kafka. It includes 29 new KIPs.
> See the release notes and release plan
> (*https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=71764913
>  >*)
> for more details. A few feature highlights:
>
> * Java 9 support with significantly faster TLS and CRC32C implementations
> (KIP)
> * JBOD improvements: disk failure only disables failed disk but not the
> broker (KIP-112/KIP-113)
> * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
> KIP-188, KIP-196)
> * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
> and drop compatibility "Evolving" annotations
>
> Release notes for the 1.0.0 release:
> *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/RELEASE_NOTES.html
> *
>
>
>
> *** Please download, test and vote by Tuesday, October 13, 8pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
> *
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> *http://home.apache.org/~guozhang/kafka-1.0.0-rc1/javadoc/
> *
>
> * Tag to be voted upon (off 1.0 branch) is the 1.0.0-rc1 tag:
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> 9424d29dbf0a3c538215b0b98b1e6b956481e4d5
>
> * Documentation:
> Note the documentation can't be pushed live due to changes that will not go
> live until the release. You can manually verify by downloading
> http://home.apache.org/~guozhang/kafka-1.0.0-rc1/
> kafka_2.11-1.0.0-site-docs.tgz
>
> * Successful Jenkins builds for the 1.0.0 branch:
> Unit/integration tests: https://builds.apache.org/job/kafka-1.0-jdk7/31/
> System test: https://jenkins.confluent.io/job/system-test-kafka-1.0/1/
>
>
> /**
>
>
> Thanks,
> -- Guozhang
>

Re: [DISCUSS] KIP-59 : Proposal for a kafka broker command

[Tom Bentley]

Hi Jayesh,

Thanks, for the KIP. I few questions/points:

1. Could you elaborate on the motivation a little? Currently it seems to
boil down to "Kafka doesn't have this, yet", but that's not, in itself, a
reason to add it. What can't be done without this change?

2. The second bullet in the "Broker information" list says "Controller
info": Is this simply whether the broker is the controller, or is there
more to it?

3. It would be better if the example program outputs each included the
command line which would produce them. Currently only the first one does.

4. Why did you choose that output format? The long lines, when wrapped
inside a terminal window, might be hard to read, and it doesn't seem very
consistent with the other tools.

5. In one place you say a '-' prefix will be used to denote trailing
partitions, and somewhere else you say '-' is used for under-replicated
partitions. I think, but am not certain, that you're using "trailing" as a
synonym for "under-replicated", and so these two sentences are saying the
same thing. "Trailing" could be taken to mean "following", so it would be
good to clarify what you mean here.

6.The KIP doesn't say so explicitly, but I think you will need to add at
least one method to the AdminClient to support this functionality. You
should describe that API, since the AdminClient is a public interface to
Kafka.

Thanks,

Tom

On 16 October 2017 at 02:44, Ted Yu  wrote:

> Please fill 'Discussion thread:' with URL to this thread.
> For 'Proposed Changes' section, is it possible to indent the lines from
> 'Broker
> Id' to 'Trailing partition count' ?
> This way, it is easier to read.
>
> bq. The command kafka-brokers.sh requires zookeeper information
>
> Is the above still true based on your updated implementation ?
>
> Thanks
>
> On Sun, Oct 15, 2017 at 6:29 PM, Jayesh Thakrar
>  > wrote:
>
> > Hi All,
> > A wiki page and development for this KIP was completed last June.However,
> > it utilized the older Admin API that used zookeeper.
> > Then I was told to hold-off until the new Admin API was ready.So this KIP
> > lay dormant since then.I have rewritten the broker command now and would
> > like to bring it up for discussion or know the next step if its not
> > discussion.
> > Here are the details:
> > KIP Wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 59%3A+Proposal+for+a+kafka+broker+command
> > Jira: https://issues.apache.org/jira/browse/KAFKA-3663
> > Pull Request: https://github.com/apache/kafka/pull/1539
> > Thank you in advance for your help/guidance.
> > Jayesh Thakrar
> >
>