[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833933#comment-15833933
 ] 

Matthias J. Sax commented on KAFKA-4144:


About #2: I guess, the question is what "dynamic" means. I interpreted it as 
"dynamic at runtime" (ie, why the application is already up an running) -- but 
maybe that is not what you meant. If you only want to have some more 
flexibility to avoid recompiling (ie, dynamic configuration at startup time) 
than this is certainly possible and desirable. 

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Matthias J. Sax]

Hi,

thanks for the KIP Eno! Here are my 2 cents:

1) I like Guozhang's proposal about removing store name from all KTable
methods and generate internal names (however, I would do this as
overloads). Furthermore, I would not force users to call .materialize()
if they want to query a store, but add one more method .stateStoreName()
that returns the store name if the KTable is materialized. Thus, also
.materialize() must not necessarily have a parameter storeName (ie, we
should have some overloads here).

I would also not allow to provide a null store name (to indicate no
materialization if not necessary) but throw an exception.

This yields some simplification (see below).


2) I also like Guozhang's proposal about KStream#toTable()


3)
> 
>>   3. What will happen when you call materialize on KTable that is already
>>   materialized? Will it create another StateStore (providing the name is
>>   different), throw an Exception?
> 
> Currently an exception is thrown, but see below.
> 
> 

If we follow approach (1) from Guozhang, there is no need to worry about
a second materialization and also no exception must be throws. A call to
.materialize() basically sets a "materialized flag" (ie, idempotent
operation) and sets a new name.


4)
>> Rename toStream() to toKStream() for consistency.
> 
> Not sure whether that is really required. We also use
> `KStreamBuilder#stream()` and `KStreamBuilder#table()`, for example, and
> don't care about the "K" prefix.

Eno's reply:
> I think changing it to `toKStream` would make it absolutely clear what we are 
> converting it to.
> 
> I'd say we should probably change the KStreamBuilder methods (but not in this 
> KIP).

I would keep #toStream(). (see below)


5) We should not remove any methods but only deprecate them.



A general note:

I do not understand your comments "Rejected Alternatives". You say "Have
the KTable be the materialized view" was rejected. But your KIP actually
does exactly this -- the changelog abstraction of KTable is secondary
after those changes and the "view" abstraction is what a KTable is. And
just to be clear, I like this a lot:

 - it aligns with the name KTable
 - is aligns with stream-table-duality
 - it aligns with IQ

I would say that a KTable is a "view abstraction" (as materialization is
optional).



-Matthias




On 1/22/17 5:05 PM, Guozhang Wang wrote:
> Thanks for the KIP Eno, I have a few meta comments and a few detailed
> comments:
> 
> 1. I like the materialize() function in general, but I would like to see
> how other KTable functions should be updated accordingly. For example, 1)
> KStreamBuilder.table(..) has a state store name parameter, and we will
> always materialize the KTable unless its state store name is set to null;
> 2) KTable.agg requires the result KTable to be materialized, and hence it
> also have a state store name; 3) KTable.join requires the joining table to
> be materialized. And today we do not actually have a mechanism to enforce
> that, but will only throw an exception at runtime if it is not (e.g. if you
> have "builder.table("topic", null).join()" a RTE will be thrown).
> 
> I'd make an extended proposal just to kick off the discussion here: let's
> remove all the state store params in other KTable functions, and if in some
> cases KTable have to be materialized (e.g. KTable resulted from KXX.agg)
> and users do not call materialize(), then we treat it as "users are not
> interested in querying it at all" and hence use an internal name generated
> for the materialized KTable; i.e. although it is materialized the state
> store is not exposed to users. And if users call materialize() afterwards
> but we have already decided to materialize it, we can replace the internal
> name with the user's provided names. Then from a user's point-view, if they
> ever want to query a KTable, they have to call materialize() with a given
> state store name. This approach has one awkwardness though, that serdes and
> state store names param are not separated and could be overlapped (see
> detailed comment #2 below).
> 
> 
> 2. This step does not need to be included in this KIP, but just as a
> reference / future work: as we have discussed before, we may enforce
> materialize KTable.join resulted KTables as well in the future. If we do
> that, then:
> 
> a) KXX.agg resulted KTables are always materialized;
> b) KTable.agg requires the aggregating KTable to always be materialized
> (otherwise we would not know the old value);
> c) KTable.join resulted KTables are always materialized, and so are the
> joining KTables to always be materialized.
> d) KTable.filter/mapValues resulted KTables materialization depend on its
> parent's materialization;
> 
> By recursive induction all KTables are actually always materialized, and
> then the effect of the "materialize()" is just for specifying the state
> store names. In this scenario, we do not need to send Change in
> repartition topics within joins any more, but only for 

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833913#comment-15833913
 ] 

Elias Levy commented on KAFKA-4144:
---

Jeyhun, sorry if I was not clear.  My comment about not being able to configure 
{{TimestampExtractor}} is only applicable to the current implementation, as it 
is instantiated by {{StreamTask}} and the current interface of 
{{TimestampExtractor}} provides not access to configuration data.  

So right now if you are consuming multiple distinct topics as different streams 
and/or tables, you have to create a case statement in your 
{{TimestampExtractor}} to handle all the topics with a single extractor, as 
that is all that you can configure right now via the {{timestamp.extractor}} 
configuration property.  And because {{TimestampExtractor}} is instantiated by 
{{StreamTask}} and the extractor does not have access to configuration data, 
you must hardcode the topic names in the extractor.  That means you can't 
change input topic names dynamically.  You can to recompile to change them.

Obviously these observation is are longer applicable if the application can 
instantiate it's own {{TimestampExtractor}}s and pass them as an argument to 
{{TopologyBuilder.addSource}}, {{KStreamBuilder.source}}, and 
{{KStreamBuilder.table}}.

As for Matthias' comments, I agree with #1 and #3.  Not quite sure what he 
means by #2.  Surely I could create a {{TimestampExtractor}} passing the 
constructor a topic name read from a config file while defining the topology, 
just like I can create SerDes that support the Confluent schema registry, where 
the registry endpoint is read from the config and use those while defining the 
topology.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833906#comment-15833906
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 54 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-02.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/23/2017 4:49:52 AM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-02.inn.local
 (10.64.37.98) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Mon, 23
 Jan 2017 03:54:33 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Mon, 23 Jan 2017 03:54:33 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd3-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd3-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id 095B4400C3
for ; Mon, 23 Jan 2017 01:54:31 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd3-us-west.apache.org (ASF Mail Server at 
spamd3-us-west.apache.org) with ESMTP id 9B5F718C458
for ; Mon, 23 Jan 2017 00:54:31 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, 
port 10024)
with ESMTP id asu2UOKP8-WJ for ;
Mon, 23 Jan 2017 00:54:30 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with 
ESMTP id C20675FC02
for ; Mon, 23 Jan 2017 00:54:29 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 7F02EE0373
for ; Mon, 23 Jan 2017 00:54:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id D02172528D
for ; Mon, 23 Jan 2017 00:54:26 + (UTC)
Date: Mon, 23 Jan 2017 00:54:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: 095B4400C3.A586E
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485737673.05479@o7lsvV/jljfPTd9NzxpItg
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-02.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-02.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833907#comment-15833907
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 54 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-02.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/23/2017 4:49:52 AM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-02.inn.local
 (10.64.37.98) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Mon, 23
 Jan 2017 03:54:35 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Mon, 23 Jan 2017 03:54:35 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd2-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd2-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id A0C65400C6
for ; Mon, 23 Jan 2017 01:54:34 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd2-us-west.apache.org (ASF Mail Server at 
spamd2-us-west.apache.org) with ESMTP id B7A441A03DC
for ; Mon, 23 Jan 2017 00:54:34 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 
10024)
with ESMTP id bVKzLCdNS7qR for ;
Mon, 23 Jan 2017 00:54:33 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with 
ESMTP id A70135FC02
for ; Mon, 23 Jan 2017 00:54:31 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 84CC5E043B
for ; Mon, 23 Jan 2017 00:54:29 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id 20726252A0
for ; Mon, 23 Jan 2017 00:54:27 + (UTC)
Date: Mon, 23 Jan 2017 00:54:27 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: A0C65400C6.A78EF
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485737675.02@7X0KwXurpfUsdUrsZoVTOA
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-02.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-02.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to this.

[jira] [Commented] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833900#comment-15833900
 ] 

Matthias J. Sax commented on KAFKA-4304:


[~jeyhunkarimov] I am not sure what you mean by {{top-k}} queries in this 
context? Currently, Interactive Queries only allow to to single key lookup and 
range queries.

> Extend Interactive Queries for return latest update timestamp per key
> -
>
> Key: KAFKA-4304
> URL: https://issues.apache.org/jira/browse/KAFKA-4304
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Priority: Minor
>  Labels: newbie++
>
> Currently, when querying state store, it is not clear when the key was 
> updated last. The ides of this JIRA is to make the latest update timestamp 
> for each key-value-pair of the state store accessible.
> For example, this might be useful to
>  * check if a value was update but did not changed (just compare the update 
> TS)
>  * if you want to consider only recently updated keys



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833899#comment-15833899
 ] 

Matthias J. Sax commented on KAFKA-4144:


Some of my thoughts:

1. I don't think, we should have a timestamp extractor per partition. A topic 
is a logical unit and should contain similar data in all partitions -- if, the 
semantics of the data in a topic should be independent of the number of 
partitions.
2. I am not sure about configuring a {{TimestampExtractor}} -- this would not 
help with dynamic topic as a configuration step would only happen once at 
startup.
3. I am also not sure about putting a {{Map}} of extractors. If you subscribe 
to multiple topics together, this indicates that those topics are "similar" and 
a single timestamp extractor should be used -- if you need different timestamp 
extractors, you just can subscribe to each topic separately.

Also keep in mind, that you can subscribe to topics by pattern -- and all 
topics matching the pattern should use the same timestamp extractor.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4113) Allow KTable bootstrap

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833891#comment-15833891
 ] 

Matthias J. Sax commented on KAFKA-4113:


[~mitch-seymour] see 
http://stackoverflow.com/questions/41601771/kafkastreams-inconsistentgroupprotocolexception
 

> Allow KTable bootstrap
> --
>
> Key: KAFKA-4113
> URL: https://issues.apache.org/jira/browse/KAFKA-4113
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>
> On the mailing list, there are multiple request about the possibility to 
> "fully populate" a KTable before actual stream processing start.
> Even if it is somewhat difficult to define, when the initial populating phase 
> should end, there are multiple possibilities:
> The main idea is, that there is a rarely updated topic that contains the 
> data. Only after this topic got read completely and the KTable is ready, the 
> application should start processing. This would indicate, that on startup, 
> the current partition sizes must be fetched and stored, and after KTable got 
> populated up to those offsets, stream processing can start.
> Other discussed ideas are:
> 1) an initial fixed time period for populating
> (it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> The API change is not decided yet, and the API desing is part of this JIRA.
> One suggestion (for option (4)) was:
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without 
> reading any other topics until see one record with timestamp 1000.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4676) Kafka consumers gets stuck for some partitions

[Vishal Shukla (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833869#comment-15833869
 ] 

Vishal Shukla commented on KAFKA-4676:
--


Hi Jason,

Thank you very much for immediate actions on this.

Consumer logs on consumer-node-01 & consumer-node-02 when topic gets stuck are 
attached in [^stuck-consumer-node-1.log] and [^consumer-node-2.log] 
respectively. This is around 2017-01-21 03:45 CET.

Config as of this time: 

{code}
session.timeout.ms=15000
max.poll.interval.ms=30
max.poll.records=500
request.timeout.ms=305
{code}

Then restarting consumer-node-02 service triggered rebalancing appropriately 
and the messages were consumed fine. Also attached the logs when triggering 
restart for both nodes as [^restart-node2-consumer-node-2.log] & 
[^restart-node2-consumer-node-1.log].

This stayed normal for few hours till around 2017-01-21 13:21 CET. This time 
the case seemed to be little different than previous case. There were no kafka 
logs in consumer-node-2. However, consumer-node-1 constantly had logs about 
rejoining, assigning partitions and warning about config as shown in 
[^stuck-case2.log].

After this case, we changed {{session.timeout.ms}} to {{30}} and 
{{max.poll.records}} to {{100}}. This gets rid of the warning and we still 
occasionally observe the rejoining & assignment logs in consumers.

{code}
2017-01-22 03:30:05,919 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking 
previously assigned partitions [] for group event-saved-group
2017-01-22 03:30:05,919 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group event-saved-group
2017-01-22 03:30:06,692 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking 
previously assigned partitions [event-saved-prod-2-8] for group 
event-saved-group
2017-01-22 03:30:06,692 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group event-saved-group
2017-01-22 03:30:06,720 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking 
previously assigned partitions [event-saved-prod-2-2] for group 
event-saved-group
2017-01-22 03:30:06,720 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group event-saved-group
2017-01-22 03:30:06,720 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking 
previously assigned partitions [event-saved-prod-2-7] for group 
event-saved-group
2017-01-22 03:30:06,720 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group event-saved-group
2017-01-22 03:30:07,956 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully 
joined group event-saved-group with generation 81
2017-01-22 03:30:07,956 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly 
assigned partitions [event-saved-prod-2-5] for group event-saved-group
2017-01-22 03:30:07,957 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully 
joined group event-saved-group with generation 81
2017-01-22 03:30:07,957 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly 
assigned partitions [event-saved-prod-2-2] for group event-saved-group
2017-01-22 03:30:07,960 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully 
joined group event-saved-group with generation 81
2017-01-22 03:30:07,960 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully 
joined group event-saved-group with generation 81
2017-01-22 03:30:07,960 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly 
assigned partitions [] for group event-saved-group
2017-01-22 03:30:07,960 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly 
assigned partitions [event-saved-prod-2-6] for group event-saved-group
2017-01-22 03:30:10,958 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking 
previously assigned partitions [event-saved-prod-2-5] for group 
event-saved-group
2017-01-22 03:30:10,958 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group event-saved-group
2017-01-22 03:30:10,971 [INFO] 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking 
previously assigned partitions [] for group event-saved-group
2017-01-22 03:30:10,971 [INFO] 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group event-saved-group
{code}

Do you see these logs as unusual? Also notice that we are running the app 
against single machine ZK & Kafka (both being in the same VM). We are aware 
that it isn't recommended and we should have ZK separately in quorum. However, 
I assume that may not 

[jira] [Updated] (KAFKA-4676) Kafka consumers gets stuck for some partitions

[Vishal Shukla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vishal Shukla updated KAFKA-4676:
-
Attachment: stuck-consumer-node-1.log
stuck-consumer-node-2.log
restart-node2-consumer-node-2.log
restart-node2-consumer-node-1.log
stuck-case2.log

> Kafka consumers gets stuck for some partitions
> --
>
> Key: KAFKA-4676
> URL: https://issues.apache.org/jira/browse/KAFKA-4676
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Vishal Shukla
>Priority: Critical
>  Labels: consumer, reliability
> Attachments: restart-node2-consumer-node-1.log, 
> restart-node2-consumer-node-2.log, stuck-case2.log, 
> stuck-consumer-node-1.log, stuck-consumer-node-2.log, 
> stuck-topic-thread-dump.log
>
>
> We recently upgraded to Kafka 0.10.1.0. We are frequently facing issue that 
> Kafka consumers get stuck suddenly for some partitions.
> Attached thread dump.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4327) Move Reset Tool from core to streams

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4327:
---
Description: 
This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008

Currently, Kafka Streams Application Reset Tool is part of {{core}} module due 
to ZK dependency. After KIP-4 got merged, this dependency can be dropped and 
the Reset Tool can be moved to {{streams}} module.

This should also update {{InternalTopicManager#filterExistingTopics}} that 
revers to ResetTool in an exception message:
{{"Use 'kafka.tools.StreamsResetter' tool"}}
-> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}

  was:
This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008

Currently, Kafka Streams Application Reset Tool is part of {{core}} module due 
to ZK dependency. After KIP-4 got merged, this dependency can be dropped and 
the Reset Tool can be moved to {{streams}} module.


> Move Reset Tool from core to streams
> 
>
> Key: KAFKA-4327
> URL: https://issues.apache.org/jira/browse/KAFKA-4327
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
>
> This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008
> Currently, Kafka Streams Application Reset Tool is part of {{core}} module 
> due to ZK dependency. After KIP-4 got merged, this dependency can be dropped 
> and the Reset Tool can be moved to {{streams}} module.
> This should also update {{InternalTopicManager#filterExistingTopics}} that 
> revers to ResetTool in an exception message:
> {{"Use 'kafka.tools.StreamsResetter' tool"}}
> -> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4613) Treat null-key records the same way for joins and aggreations

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4613:
---
Status: Patch Available  (was: Open)

> Treat null-key records the same way for joins and aggreations
> -
>
> Key: KAFKA-4613
> URL: https://issues.apache.org/jira/browse/KAFKA-4613
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
>
> Currently, on aggregation records with null-key get dropped while for joins 
> we raise an exception.
> We might want to drop in both cases of raise an exception in both cases to be 
> consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3835) Streams is creating two ProducerRecords for each send via RecordCollector

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833786#comment-15833786
 ] 

Guozhang Wang commented on KAFKA-3835:
--

[~ankushpuri] Sorry for the late reply as I was not pinged and did not get a 
notification. I have added you to the contribution list so you can try assign 
tasks to yourself.

Please take a look at the current newbie JIRAs and see if there are any other 
tasks that you are interested, and also feel free to un-assign yourself if you 
are busy and would like others to pick them up: 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20%3D%20Open%20AND%20labels%20%3D%20newbie

> Streams is creating two ProducerRecords for each send via RecordCollector
> -
>
> Key: KAFKA-3835
> URL: https://issues.apache.org/jira/browse/KAFKA-3835
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Damian Guy
>Assignee: Jeyhun Karimov
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.3.0
>
>
> The RecordCollector.send(..) method below, currently receives a 
> ProducerRecord from its caller and then creates another one to forward on to 
> its producer.  The creation of 2 ProducerRecords should be removed.
> {code}
> public  void send(ProducerRecord record, Serializer 
> keySerializer, Serializer valueSerializer,
> StreamPartitioner partitioner)
> {code}
> We could replace the above method with
> {code}
> public  void send(String topic,
> K key,
> V value,
> Integer partition,
> Long timestamp,
> Serializer keySerializer,
> Serializer valueSerializer,
> StreamPartitioner partitioner)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1850) Failed reassignment leads to additional replica

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1850:
-
Assignee: (was: Neha Narkhede)

> Failed reassignment leads to additional replica
> ---
>
> Key: KAFKA-1850
> URL: https://issues.apache.org/jira/browse/KAFKA-1850
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8.1
> Environment: CentOS  (Linux Kernel 2.6.32-71.el6.x86_64 )
>Reporter: Alex Tian
>Priority: Minor
>  Labels: newbie
> Attachments: 
> track_on_testingTopic-9_movement_on_the_following_2_days.txt, Track on 
> testingTopic-9's movement.txt
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> When I start a topic reassignment (Totally 36 partitions) in my Kafka 
> Cluster, 24 partitions succeeded and 12 ones failed. However, the 12 failed 
> partitions have more replicas. I think the reason is that  AR still consists 
> of RAR and OAR although the reassignment for the partition failed. Could we 
> regard this problem as a bug? Quite sorry if any mistake in my question, 
> since I am a beginner for Kafka.
> This is the output from operation: 
> 1. alex-topics-to-move.json:
> {"topics": [{"topic": "testingTopic"}],
>  "version":1
> }
> 2. Generate a reassignment plan
> $./kafka-reassign-partitions.sh  --generate  --broker-list 0,1,2,3,4 
> --topics-to-move-json-file ./alex-topics-to-move.json   --zookeeper 
> 192.168.112.95:2181,192.168.112.96:2181,192.168.112.97:2181,192.168.112.98:2181,192.168.112.99:2181
> Current partition replica assignment
> {"version":1,
>  "partitions":[   {"topic":"testingTopic","partition":27,"replicas":[0,2]},
>
> {"topic":"testingTopic","partition":1,"replicas":[1,2]},
>   {"topic":"testingTopic","partition":12,"replicas":[0,1]},
>   {"topic":"testingTopic","partition":6,"replicas":[0,1]},
>   {"topic":"testingTopic","partition":16,"replicas":[1,0]},
>   {"topic":"testingTopic","partition":32,"replicas":[2,0]},
>   {"topic":"testingTopic","partition":18,"replicas":[0,1]},
>   {"topic":"testingTopic","partition":31,"replicas":[1,2]},
>   {"topic":"testingTopic","partition":9,"replicas":[0,2]},
>   {"topic":"testingTopic","partition":23,"replicas":[2,1]},
>   {"topic":"testingTopic","partition":19,"replicas":[1,2]},
>   {"topic":"testingTopic","partition":34,"replicas":[1,0]},
>   {"topic":"testingTopic","partition":17,"replicas":[2,1]},
>   {"topic":"testingTopic","partition":7,"replicas":[1,2]},
>   {"topic":"testingTopic","partition":20,"replicas":[2,0]},
>   {"topic":"testingTopic","partition":8,"replicas":[2,0]},
>   {"topic":"testingTopic","partition":11,"replicas":[2,1]},
>   {"topic":"testingTopic","partition":3,"replicas":[0,2]},
>   {"topic":"testingTopic","partition":30,"replicas":[0,1]},
>   {"topic":"testingTopic","partition":35,"replicas":[2,1]},
>   {"topic":"testingTopic","partition":26,"replicas":[2,0]},
>   {"topic":"testingTopic","partition":22,"replicas":[1,0]},
>   {"topic":"testingTopic","partition":10,"replicas":[1,0]},
>   {"topic":"testingTopic","partition":24,"replicas":[0,1]},
>   {"topic":"testingTopic","partition":21,"replicas":[0,2]},
>   {"topic":"testingTopic","partition":15,"replicas":[0,2]},
>   {"topic":"testingTopic","partition":4,"replicas":[1,0]},
>   {"topic":"testingTopic","partition":28,"replicas":[1,0]},
>   {"topic":"testingTopic","partition":25,"replicas":[1,2]},:
>   {"topic":"testingTopic","partition":14,"replicas":[2,0]},
>   {"topic":"testingTopic","partition":2,"replicas":[2,0]},
>   {"topic":"testingTopic","partition":13,"replicas":[1,2]},
>   {"topic":"testingTopic","partition":5,"replicas":[2,1]},
>   {"topic":"testingTopic","partition":29,"replicas":[2,1]},
>   {"topic":"testingTopic","partition":33,"replicas":[0,2]},
>   {"topic":"testingTopic","partition":0,"replicas":[0,1]}]}
>  Proposed partition reassignment configuration  ( 
> alex-expand-cluster-reassignment.json )
> {"version":1,
>  "partitions":[   
> {"topic":"testingTopic","partition":27,"replicas":[0,4]},
>{"topic":"testingTopic","partition":1,"replicas":[4,2]},
>
> 

[jira] [Commented] (KAFKA-1354) Failed to load class "org.slf4j.impl.StaticLoggerBinder"

[Maniyoor, RakeshAcharya (Rakesh) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833784#comment-15833784
 ] 

Maniyoor, RakeshAcharya (Rakesh) commented on KAFKA-1354:
-

HI

I am currently OOO and have limited access to mails , So will respond once am 
back.


Cheers

Rakesh

This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.


> Failed to load class "org.slf4j.impl.StaticLoggerBinder"
> 
>
> Key: KAFKA-1354
> URL: https://issues.apache.org/jira/browse/KAFKA-1354
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: RHEL
>Reporter: RakeshAcharya
>  Labels: newbie, patch, usability
> Fix For: 0.10.2.0
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Getting below errors during Kafka startup
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
> details.
> [2014-03-31 18:55:36,488] INFO Will not load MX4J, mx4j-tools.jar is not in 
> the classpath (kafka.utils.Mx4jLoader$)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2363) ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread Transient Failure

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833785#comment-15833785
 ] 

Guozhang Wang commented on KAFKA-2363:
--

I have not seen this failure recently, how about you [~ijuma]?

> ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread Transient Failure
> ---
>
> Key: KAFKA-2363
> URL: https://issues.apache.org/jira/browse/KAFKA-2363
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Fangmin Lv
>Assignee: Ben Stopford
>  Labels: newbie, transient-unit-test-failure
> Fix For: 0.10.2.0
>
>
> {code}
> kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
> STANDARD_OUT
> [2015-07-24 23:13:05,148] WARN fsync-ing the write ahead log in SyncThread:0 
> took 1084ms which will adversely effect operation latency. See the ZooKeeper 
> troubleshooting guide (org.apache.zookeeper.s
> erver.persistence.FileTxnLog:334)
> kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromCallerThread FAILED
> java.lang.AssertionError: No request is complete.
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:44)
> at 
> kafka.api.ProducerSendTest$$anonfun$testCloseWithZeroTimeoutFromCallerThread$1.apply$mcVI$sp(ProducerSendTest.scala:343)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at 
> kafka.api.ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread(ProducerSendTest.scala:340)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1676) Ability to cancel replica reassignment in progress

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1676:
-
Assignee: (was: Neha Narkhede)

> Ability to cancel replica reassignment in progress
> --
>
> Key: KAFKA-1676
> URL: https://issues.apache.org/jira/browse/KAFKA-1676
> Project: Kafka
>  Issue Type: New Feature
>  Components: controller
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
>
> I've had several situations where I have started a replica reassignment that 
> I've needed to cancel before it completed.
> This has happened 
> * when moving to a new broker that turns out to be running on an impaired 
> server
> * if the extra replication hurts cluster performance
> * dealing with replication bugs in kafka, like KAFKA-1670
> * when a single replica reassignment is taking a long time, and I want to 
> start more replica assignments without waiting for the current one to finish.
> For the first three cases, as a last resort I have deleted the 
> {{/admin/reassign_partitions}} key from ZooKeeper and restarted the 
> controller. I would like to be able to do this by signaling the controller to 
> stop, and to leave the list of assignments as they exist at that moment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3124) Update protocol wiki page to reflect latest request/response formats

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3124:
-
Assignee: (was: Liquan Pei)

> Update protocol wiki page to reflect latest request/response formats
> 
>
> Key: KAFKA-3124
> URL: https://issues.apache.org/jira/browse/KAFKA-3124
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Joel Koshy
>  Labels: newbie
>
> The protocol wiki 
> (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)
>  is slightly out of date. It does not have some of the newer request/response 
> formats.
> We should actually figure out a way to _source_ the protocol definitions from 
> the last official release version into that wiki.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1354) Failed to load class "org.slf4j.impl.StaticLoggerBinder"

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1354:
-
Assignee: (was: Jay Kreps)

> Failed to load class "org.slf4j.impl.StaticLoggerBinder"
> 
>
> Key: KAFKA-1354
> URL: https://issues.apache.org/jira/browse/KAFKA-1354
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: RHEL
>Reporter: RakeshAcharya
>  Labels: newbie, patch, usability
> Fix For: 0.10.2.0
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Getting below errors during Kafka startup
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
> details.
> [2014-03-31 18:55:36,488] INFO Will not load MX4J, mx4j-tools.jar is not in 
> the classpath (kafka.utils.Mx4jLoader$)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3124) Update protocol wiki page to reflect latest request/response formats

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833783#comment-15833783
 ] 

Guozhang Wang commented on KAFKA-3124:
--

[~liquanpei] I'm un-assigning you from this task for now, please feel free to 
re pick them up when you have time.

> Update protocol wiki page to reflect latest request/response formats
> 
>
> Key: KAFKA-3124
> URL: https://issues.apache.org/jira/browse/KAFKA-3124
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Joel Koshy
>Assignee: Liquan Pei
>  Labels: newbie
>
> The protocol wiki 
> (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)
>  is slightly out of date. It does not have some of the newer request/response 
> formats.
> We should actually figure out a way to _source_ the protocol definitions from 
> the last official release version into that wiki.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Guozhang Wang]

Thanks for the KIP Eno, I have a few meta comments and a few detailed
comments:

1. I like the materialize() function in general, but I would like to see
how other KTable functions should be updated accordingly. For example, 1)
KStreamBuilder.table(..) has a state store name parameter, and we will
always materialize the KTable unless its state store name is set to null;
2) KTable.agg requires the result KTable to be materialized, and hence it
also have a state store name; 3) KTable.join requires the joining table to
be materialized. And today we do not actually have a mechanism to enforce
that, but will only throw an exception at runtime if it is not (e.g. if you
have "builder.table("topic", null).join()" a RTE will be thrown).

I'd make an extended proposal just to kick off the discussion here: let's
remove all the state store params in other KTable functions, and if in some
cases KTable have to be materialized (e.g. KTable resulted from KXX.agg)
and users do not call materialize(), then we treat it as "users are not
interested in querying it at all" and hence use an internal name generated
for the materialized KTable; i.e. although it is materialized the state
store is not exposed to users. And if users call materialize() afterwards
but we have already decided to materialize it, we can replace the internal
name with the user's provided names. Then from a user's point-view, if they
ever want to query a KTable, they have to call materialize() with a given
state store name. This approach has one awkwardness though, that serdes and
state store names param are not separated and could be overlapped (see
detailed comment #2 below).


2. This step does not need to be included in this KIP, but just as a
reference / future work: as we have discussed before, we may enforce
materialize KTable.join resulted KTables as well in the future. If we do
that, then:

a) KXX.agg resulted KTables are always materialized;
b) KTable.agg requires the aggregating KTable to always be materialized
(otherwise we would not know the old value);
c) KTable.join resulted KTables are always materialized, and so are the
joining KTables to always be materialized.
d) KTable.filter/mapValues resulted KTables materialization depend on its
parent's materialization;

By recursive induction all KTables are actually always materialized, and
then the effect of the "materialize()" is just for specifying the state
store names. In this scenario, we do not need to send Change in
repartition topics within joins any more, but only for repartitions topics
within aggregations. Instead, we can just send a "tombstone" without the
old value and we do not need to calculate joins twice (one more time when
old value is received).

3. I'm wondering if it is worth-while to add a "KStream#toTable()" function
which is interpreted as a dummy-aggregation where the new value always
replaces the old value. I have seen a couple of use cases of this, for
example, users want to read a changelog topic, apply some filters, and then
materialize it into a KTable with state stores without creating duplicated
changelog topics. With materialize() and toTable I'd imagine users can
specify sth. like:

"
KStream stream = builder.stream("topic1").filter(..);
KTable table = stream.toTable(..);
table.materialize("state1");
"

And the library in this case could set store "state1" 's changelog topic to
be "topic1", and applying the filter on the fly while (re-)storing its
state by reading from this topic, instead of creating a second changelog
topic like "appID-state1-changelog" which is a semi-duplicate of "topic1".


Detailed:

1. I'm +1 with Michael regarding "#toStream"; actually I was thinking about
renaming to "#toChangeLog" but after thinking a bit more I think #toStream
is still better, and we can just mention in the javaDoc that it is
transforming its underlying changelog stream to a normal stream.
2. As Damian mentioned, there are a few scenarios where the serdes are
already specified in a previous operation whereas it is not known before
calling materialize, for example:
stream.groupByKey.agg(serde).materialize(serde) v.s. table.mapValues(/*no
serde specified*/).materialize(serde). We need to specify what are the
handling logic here.
3. We can remove "KTable#to" call as well, and enforce users to call "
KTable.toStream.to" to be more clear.


Guozhang


On Wed, Jan 18, 2017 at 3:22 AM, Eno Thereska 
wrote:

> I think changing it to `toKStream` would make it absolutely clear what we
> are converting it to.
>
> I'd say we should probably change the KStreamBuilder methods (but not in
> this KIP).
>
> Thanks
> Eno
>
> > On 17 Jan 2017, at 13:59, Michael Noll  wrote:
> >
> >> Rename toStream() to toKStream() for consistency.
> >
> > Not sure whether that is really required. We also use
> > `KStreamBuilder#stream()` and `KStreamBuilder#table()`, for example, and
> > don't care about the "K" prefix.
> >
> >
> >
> > On Tue, Jan 17, 2017 at 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833775#comment-15833775
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 51 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/23/2017 12:43:01 AM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 23:44:35 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 23:44:35 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd1-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd1-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id A054C400C3
for ; Sun, 22 Jan 2017 21:44:34 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd1-us-west.apache.org (ASF Mail Server at 
spamd1-us-west.apache.org) with ESMTP id 4DECEC0981
for ; Sun, 22 Jan 2017 20:44:34 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 
10024)
with ESMTP id f2-di-WcVpre for ;
Sun, 22 Jan 2017 20:44:30 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with 
ESMTP id 912225F479
for ; Sun, 22 Jan 2017 20:44:29 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 838E0E0362
for ; Sun, 22 Jan 2017 20:44:27 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id B166F2528D
for ; Sun, 22 Jan 2017 20:44:26 + (UTC)
Date: Sun, 22 Jan 2017 20:44:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: A054C400C3.A656F
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485722675.33512@J5I6e2R7VVPaems8QZKWxw
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833776#comment-15833776
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 51 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/23/2017 12:43:01 AM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 23:44:40 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 23:44:40 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd2-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd2-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id 7C041400C3
for ; Sun, 22 Jan 2017 21:44:39 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd2-us-west.apache.org (ASF Mail Server at 
spamd2-us-west.apache.org) with ESMTP id 4EEB41A032E
for ; Sun, 22 Jan 2017 20:44:39 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 
10024)
with ESMTP id 5btBNWUjZsCS for ;
Sun, 22 Jan 2017 20:44:35 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with 
ESMTP id 4A95F60E83
for ; Sun, 22 Jan 2017 20:44:33 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 6271CE0419
for ; Sun, 22 Jan 2017 20:44:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id 04501252A1
for ; Sun, 22 Jan 2017 20:44:27 + (UTC)
Date: Sun, 22 Jan 2017 20:44:27 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: 7C041400C3.A6EF1
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485722680.05295@Z9DJh9ecbJtY/gBi269IZA
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Dong Lin]

Hey Guozhang,

Thanks for the review! Yes we have considered this approach and briefly
explained why we don't do it in the rejected alternative section. Here is
my concern with this approach in more detail:

- This approach introduces tight coupling between kafka's logical leader
election with broker's local file OS config. My intuition is that this
tight coupling may make future development a bit harder and we should try
to avoid that. Note that we only use logical information (e.g. partition,
broker id) in the zookeeper and controller as of now.

- Encoding log directory in the replica identifier requires much more
change in the code. In addition to changing znode data format in zookeeper,
we probably need to update every protocol that touches replica id, such as
StopReplicaRequest, ListOffsetRequest, LeaderAndIsrResponse and so on. Many
Java classes need to be changes as well to recognize log directory in
replica identifier. Arguably it is still possible to use broker id without
log directory to identify replica in some protocols and Java classes under
the assumption that no two replicas of the same partition can reside on the
same broker. But we need to think carefully for each protocol and Java
class and the result may be error prone and controversial. For simplicity
of the discussion and code review, I prefer to only do this if there is
strong benefit of this design.

- Current approach in the KIP make it easier to move replicas between
replicas on the same broker because that operation can be completely hidden
from controller and other brokers. On the other hand, if we were to move
replica between disk in the suggested approach, broker needs to write to
some notification zookeeper path after movement is completed so that broker
can send LeaderAndIsrRequest to get the new replica identifier, update it
cache and write to znode /brokers/topics/[topic]/partitions/[partitionId]/
state.

Dong


On Sun, Jan 22, 2017 at 10:50 AM, Guozhang Wang  wrote:

> Hello Dong,
>
> Thanks for the very well written KIP. I had a general thought on the ZK
> path management, wondering if the following alternative would work:
>
> 1. Bump up versions in "brokers/topics/[topic]" and
> "/brokers/topics/[topic]/partitions/[partitionId]/state"
> to 2, in which the replica id is no longer an int but a string.
>
> 2. Bump up versions in "/brokers/ids/[brokerId]" to add another field:
>
> { "fields":
> [ {"name": "version", "type": "int", "doc": "version id"},
>   {"name": "host", "type": "string", "doc": "ip address or host name of
> the broker"},
>   {"name": "port", "type": "int", "doc": "port of the broker"},
>   {"name": "jmx_port", "type": "int", "doc": "port for jmx"}
>   {"name": "log_dirs",
>"type": {"type": "array",
> "items": "int",
> "doc": "an array of the id of the log dirs in broker"}
>   },
> ]
> }
>
> 3. The replica id can now either be an string-typed integer indicating that
> all partitions on the broker still treated as failed or not as a whole,
> i.e. no support needed for JBOD; or be a string typed "[brokerID]-[dirID]",
> in which brokers / controllers can still parse to determine which broker is
> hosting this replica: in this case the management of replicas is finer
> grained, no longer at the broker level (i.e. if broker dies all replicas go
> offline) but broker-dir level.
>
> 4. When broker had one of the dir failed, it can modify its "
> /brokers/ids/[brokerId]" registry and remove the dir id, controller already
> listening on this path can then be notified and run the replica assignment
> accordingly where replica id is computed as above.
>
>
> By doing this controller can also naturally reassign replicas between dirs
> within the same broker.
>
>
> Guozhang
>
>
> On Thu, Jan 12, 2017 at 6:25 PM, Ismael Juma  wrote:
>
> > Thanks for the KIP. Just wanted to quickly say that it's great to see
> > proposals for improving JBOD (KIP-113 too). More feedback soon,
> hopefully.
> >
> > Ismael
> >
> > On Thu, Jan 12, 2017 at 6:46 PM, Dong Lin  wrote:
> >
> > > Hi all,
> > >
> > > We created KIP-112: Handle disk failure for JBOD. Please find the KIP
> > wiki
> > > in the link https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 112%3A+Handle+disk+failure+for+JBOD.
> > >
> > > This KIP is related to KIP-113
> > >  > > 113%3A+Support+replicas+movement+between+log+directories>:
> > > Support replicas movement between log directories. They are needed in
> > order
> > > to support JBOD in Kafka. Please help review the KIP. You feedback is
> > > appreciated!
> > >
> > > Thanks,
> > > Dong
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Dong Lin]

Thanks for the comment Guozhang. Please don't worry about being late. I
would like to update the KIP if there is clear benefit of the new approach.
I am wondering if there is any use-case or operation aspects that would
benefit from the new approach.

I am not saying that these checkpoint files have the same priority. I
mentioned other checkpoint files to suggest that it is OK to add one more
checkpoint file. To me three checkpoint files is not much different from
four checkpoint files. I am just inclined to not update the KIP if the only
benefit is to avoid addition of a new checkpoint file.



On Sun, Jan 22, 2017 at 3:48 PM, Guozhang Wang  wrote:

> To me the distinction between recovery-checkpoint and
> replication-checkpoint are different from the distinction between these two
> hw checkpoint values: when broker starts up and act as the leader for a
> partition, it can live without seeing the recovery checkpoint, but just
> cannot rely on the existing last log segment and need to fetch from other
> replicas; but if the replication-checkpoint file is missing, it is a
> correctness issue, as it does not know from where to truncate its data, and
> also how to respond to a fetch request. That is why I think we can separate
> these two types of files, since the latter one is more important than the
> previous one.
>
> That being said, I do not want to recall another vote on this since it is
> my bad not responding before the vote is called. Just wanted to point out
> for the record that this approach may have some operational scenarios where
> one of the replication files is missing and we need to treat them
> specifically.
>
>
> Guozhang
>
>
> On Sun, Jan 22, 2017 at 1:56 PM, Dong Lin  wrote:
>
> > Yeah, your solution of adding new APIs certainly works and I don't think
> > that is an issue. On the other hand I don't think it is an issue to add a
> > new checkpoint file as well since we already have multiple checkpoint
> > files. The benefit of the new approach you mentioned is probably not an
> > issue in the current approach since high watermark and low watermark
> works
> > completely independently. Since there is no strong reason to choose
> either
> > of them, I am inclined to choose the one that makes less format change
> and
> > simpler in the Java API. The current approach seems better w.r.t this
> minor
> > reason.
> >
> > If you are strong that we should use the new approach, I can do that as
> > well. Please let me know if you think so, and I will need to ask
> > Jun/Joel/Becket to vote on this again since this changes the interface of
> > the KIP.
> >
> > On Sun, Jan 22, 2017 at 9:35 AM, Guozhang Wang 
> wrote:
> >
> > > I think this is less of an issue: we can use the same patterns as in
> the
> > > request protocol, i.e.:
> > >
> > > write(Map[TP, Long]) // write the checkout point in v0 format
> > > write(Map[TP, Pair[Long, Long]]) // write the checkout point in v1
> format
> > >
> > > CheckpointedOffsets read() // read the file relying on its version id
> > >
> > > class CheckpointedOffsets {
> > >
> > > Integer getVersion();
> > > Long getFirstOffset();
> > > Long getSecondOffset();   // would return NO_AVAILABLE with v0
> format
> > > }
> > >
> > >
> > > As I think of it, another benefit is that we wont have a partition that
> > > only have one of the watermarks in case of a failure in between writing
> > two
> > > files.
> > >
> > > Guozhang
> > >
> > > On Sun, Jan 22, 2017 at 12:03 AM, Dong Lin 
> wrote:
> > >
> > > > Hey Guozhang,
> > > >
> > > > Thanks for the review:) Yes it is possible to combine them. Both
> > solution
> > > > will have the same performance. But I think the current solution will
> > > give
> > > > us simpler Java class design. Note that we will have to change Java
> API
> > > > (e.g. read() and write()) of OffsetCheckpoint class in order to
> > provide a
> > > > map from TopicPartition to a pair of integers when we write to
> > checkpoint
> > > > file. This makes this class less generic since this API is not used
> by
> > > log
> > > > recovery checkpoint and log cleaner checkpoint which are also using
> > > > OffsetCheckpoint class.
> > > >
> > > > Dong
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sat, Jan 21, 2017 at 12:28 PM, Guozhang Wang 
> > > > wrote:
> > > >
> > > > > Hi Dong,
> > > > >
> > > > > Sorry for being late on reviewing this KIP. It LGTM overall, but
> I'm
> > > > > wondering if we can save adding the "replication-low-watermark-
> > > > checkpoint"
> > > > > file by just bumping up the version number of "replication-offset-
> > > > > checkpoint"
> > > > > to let it have two values for each partition, i.e.:
> > > > >
> > > > > 1  // version number
> > > > > [number of partitions]
> > > > > [topic name] [partition id] [lwm] [hwm]
> > > > >
> > > > >
> > > > > This will affects the upgrade path a bit, but I think 

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Guozhang Wang]

To me the distinction between recovery-checkpoint and
replication-checkpoint are different from the distinction between these two
hw checkpoint values: when broker starts up and act as the leader for a
partition, it can live without seeing the recovery checkpoint, but just
cannot rely on the existing last log segment and need to fetch from other
replicas; but if the replication-checkpoint file is missing, it is a
correctness issue, as it does not know from where to truncate its data, and
also how to respond to a fetch request. That is why I think we can separate
these two types of files, since the latter one is more important than the
previous one.

That being said, I do not want to recall another vote on this since it is
my bad not responding before the vote is called. Just wanted to point out
for the record that this approach may have some operational scenarios where
one of the replication files is missing and we need to treat them
specifically.


Guozhang


On Sun, Jan 22, 2017 at 1:56 PM, Dong Lin  wrote:

> Yeah, your solution of adding new APIs certainly works and I don't think
> that is an issue. On the other hand I don't think it is an issue to add a
> new checkpoint file as well since we already have multiple checkpoint
> files. The benefit of the new approach you mentioned is probably not an
> issue in the current approach since high watermark and low watermark works
> completely independently. Since there is no strong reason to choose either
> of them, I am inclined to choose the one that makes less format change and
> simpler in the Java API. The current approach seems better w.r.t this minor
> reason.
>
> If you are strong that we should use the new approach, I can do that as
> well. Please let me know if you think so, and I will need to ask
> Jun/Joel/Becket to vote on this again since this changes the interface of
> the KIP.
>
> On Sun, Jan 22, 2017 at 9:35 AM, Guozhang Wang  wrote:
>
> > I think this is less of an issue: we can use the same patterns as in the
> > request protocol, i.e.:
> >
> > write(Map[TP, Long]) // write the checkout point in v0 format
> > write(Map[TP, Pair[Long, Long]]) // write the checkout point in v1 format
> >
> > CheckpointedOffsets read() // read the file relying on its version id
> >
> > class CheckpointedOffsets {
> >
> > Integer getVersion();
> > Long getFirstOffset();
> > Long getSecondOffset();   // would return NO_AVAILABLE with v0 format
> > }
> >
> >
> > As I think of it, another benefit is that we wont have a partition that
> > only have one of the watermarks in case of a failure in between writing
> two
> > files.
> >
> > Guozhang
> >
> > On Sun, Jan 22, 2017 at 12:03 AM, Dong Lin  wrote:
> >
> > > Hey Guozhang,
> > >
> > > Thanks for the review:) Yes it is possible to combine them. Both
> solution
> > > will have the same performance. But I think the current solution will
> > give
> > > us simpler Java class design. Note that we will have to change Java API
> > > (e.g. read() and write()) of OffsetCheckpoint class in order to
> provide a
> > > map from TopicPartition to a pair of integers when we write to
> checkpoint
> > > file. This makes this class less generic since this API is not used by
> > log
> > > recovery checkpoint and log cleaner checkpoint which are also using
> > > OffsetCheckpoint class.
> > >
> > > Dong
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Sat, Jan 21, 2017 at 12:28 PM, Guozhang Wang 
> > > wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > Sorry for being late on reviewing this KIP. It LGTM overall, but I'm
> > > > wondering if we can save adding the "replication-low-watermark-
> > > checkpoint"
> > > > file by just bumping up the version number of "replication-offset-
> > > > checkpoint"
> > > > to let it have two values for each partition, i.e.:
> > > >
> > > > 1  // version number
> > > > [number of partitions]
> > > > [topic name] [partition id] [lwm] [hwm]
> > > >
> > > >
> > > > This will affects the upgrade path a bit, but I think not by large,
> and
> > > all
> > > > other logic will not be affected.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > > On Wed, Jan 18, 2017 at 6:12 PM, Dong Lin 
> wrote:
> > > >
> > > > > Thanks to everyone who voted and provided feedback!
> > > > >
> > > > > This KIP is now adopted with 3 binding +1s (Jun, Joel, Becket) and
> 2
> > > > > non-binding +1s (Radai, Mayuresh).
> > > > >
> > > > > Thanks,
> > > > > Dong
> > > > >
> > > > > On Wed, Jan 18, 2017 at 6:05 PM, Jun Rao  wrote:
> > > > >
> > > > > > Hi, Dong,
> > > > > >
> > > > > > Thanks for the update. +1
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Jan 18, 2017 at 1:44 PM, Dong Lin 
> > > wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > After some more thinking, I agree with you that it is better to
> > > > simply
> > > > > > > 

[jira] [Commented] (KAFKA-4113) Allow KTable bootstrap

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833762#comment-15833762
 ] 

Guozhang Wang commented on KAFKA-4113:
--

Hello [~mitch-seymour], sorry for the late reply. In a recent change we add 
overloaded functions in {{builder.table/stream}} calls so that users can 
specify different "reset policies" per stream. For example, you can do the 
following:

{code}

myTable = builder.table(EARLEST, "topic1", ..);
myStream = builder.stream(LATEST, "topic2", ..);

myStream.join(myTable)...

{code}

If there is no committed offsets (i.e. the application has not executed 
before), then different streams will use their own reset policies; if it is not 
specified then the global config value in {{ConsumerConfig}} is used.

Does this solve your use case?

> Allow KTable bootstrap
> --
>
> Key: KAFKA-4113
> URL: https://issues.apache.org/jira/browse/KAFKA-4113
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>
> On the mailing list, there are multiple request about the possibility to 
> "fully populate" a KTable before actual stream processing start.
> Even if it is somewhat difficult to define, when the initial populating phase 
> should end, there are multiple possibilities:
> The main idea is, that there is a rarely updated topic that contains the 
> data. Only after this topic got read completely and the KTable is ready, the 
> application should start processing. This would indicate, that on startup, 
> the current partition sizes must be fetched and stored, and after KTable got 
> populated up to those offsets, stream processing can start.
> Other discussed ideas are:
> 1) an initial fixed time period for populating
> (it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> The API change is not decided yet, and the API desing is part of this JIRA.
> One suggestion (for option (4)) was:
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without 
> reading any other topics until see one record with timestamp 1000.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2407: logDirs empty error added

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2407


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Suggestion] Issues on StreamConfig Builder

[Guozhang Wang]

Sounds good, thanks!

On Thu, Jan 19, 2017 at 9:21 PM, Dongjin Lee  wrote:

> 1. Then, I will submit the KIP as soon as KAFKA-2898 is completed.
> 2. Into following three sub-tasks:
> a. Provide ProducerConfig#builder
> b. Provide ConsumerConfig#builder
> c. Provide StreamsConfig#builder on top of a and b.
>
> Thanks,
> Dongjin
>
> On Fri, Jan 20, 2017 at 1:20 PM, Guozhang Wang  wrote:
>
> > 1. I think it would need a KIP discussion, yes.
> > 2. What sub-tasks are you considering to break KAFKA-4436?
> >
> >
> > Guozhang
> >
> > On Thu, Jan 19, 2017 at 5:52 PM, Dongjin Lee  wrote:
> >
> > > Guozhang and all,
> > >
> > > No problem. I just took KAFKA-2898. Let's concentrate on KAFKA-4436:
> > >
> > > 1. Does this issue require KIP? If it is true, I will prepare the KIP.
> > > 2. Should this issue be moved into Umbrella task, with separated
> > sub-tasks?
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Fri, Jan 20, 2017 at 2:22 AM, Guozhang Wang 
> > wrote:
> > >
> > >> Dongjin,
> > >>
> > >> Thanks for your interests. I looked at these three tickets, and I
> think
> > we
> > >> can tackle them incrementally instead of getting them in one shot,
> that
> > is
> > >> because:
> > >>
> > >> 1. Although I see some dependencies between 4436 and 2898, I think
> they
> > >> can
> > >> still be tackled separately (though like you said, probably better
> > >> tackling
> > >> 2898 then consider doing 4436 which would likely require a KIP
> > discussion
> > >> and we may want to do that for all clients, not only Streams).
> > >>
> > >> 2. Vahid seems already actively working on 4531, which I think is a
> bit
> > >> orthogonal to other two.
> > >>
> > >> 3. We prefer to work on smaller tasks, which generates smaller patches
> > to
> > >> review etc.
> > >>
> > >>
> > >> So how about start picking on 2898 first, which I think can be done in
> > >> parallel with 4531?
> > >>
> > >> Guozhang
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Jan 18, 2017 at 12:31 AM, Dongjin Lee 
> > wrote:
> > >>
> > >> > Hello. This mail is related to KAFKA-4436[^1], which Matthias left a
> > >> > comment recently. I started to investigate this issue from last
> > weekend,
> > >> > and I thought that it would be much better to take your advice.
> > >> >
> > >> > What I would like to talk about this issue is following two things:
> > >> >
> > >> > 1. This issue is much larger than expected. To support
> > >> > StreamsConfig#builder, we have to implement builders for
> > ProducerConfig
> > >> and
> > >> > ConsumerConfig first. We can start this job only after implementing
> > and
> > >> > validating all the methods of ProducerConfig and ConsumerConfig.
> > >> > 2. Some other config validation issues are related to this issue. I
> > >> already
> > >> > found following two, but it seems like there are more related ones:
> > >> > - KAFKA-2898: Kafka producer config incorrectly warns that
> > >> serializer.*
> > >> > isn't a known config[^2]
> > >> > - KAFKA-4531: Rationalise client configuration validation[^3]
> > >> >
> > >> > So, I suggest:
> > >> >
> > >> > 1. Change KAFKA-4436 into umbrella issue and add ProducerConfig,
> > >> > ConsumerConfig and StreamsConfig as separate sub-issues.
> > >> > 2. Assign above issues to me. I will take all the issues - and if
> you
> > >> find
> > >> > any validation issues like KAFKA-2898 or KAFKA-4531, don't hesitate
> to
> > >> > leave a comment or send me a message. I will also handle them with
> the
> > >> > config builders.
> > >> > 3. If needed, I will submit the KIP for this issue.
> > >> >
> > >> > Thanks for reading. I am looking forward to your comments.
> > >> >
> > >> > Best,
> > >> > Dongjin
> > >> >
> > >> > [^1]: https://issues.apache.org/jira/browse/KAFKA-4436
> > >> > [^2]: https://issues.apache.org/jira/browse/KAFKA-2898
> > >> > [^3]: https://issues.apache.org/jira/browse/KAFKA-4531
> > >> >
> > >> > --
> > >> > *Dongjin Lee*
> > >> >
> > >> >
> > >> > *Software developer in Line+.So interested in massive-scale machine
> > >> > learning.facebook: www.facebook.com/dongjin.lee.kr
> > >> > linkedin:
> > >> > kr.linkedin.com/in/dongjinleekr
> > >> > github:
> > >> > github.com/dongjinleekr
> > >> > twitter:
> www.twitter.com/dongjinleekr
> > >> > *
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > >
> > > *Software developer in Line+.So interested in massive-scale machine
> > > learning.facebook: www.facebook.com/dongjin.lee.kr
> > > linkedin: kr.linkedin.com/in/
> > dongjinleekr
> > > github:
> > > github.com/dongjinleekr
> > > 

[jira] [Commented] (KAFKA-4676) Kafka consumers gets stuck for some partitions

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833747#comment-15833747
 ] 

Jason Gustafson commented on KAFKA-4676:


Can you provide full consumer logs please? It is also helpful if you can 
identify which partitions were stuck and at what time it happened.

> Kafka consumers gets stuck for some partitions
> --
>
> Key: KAFKA-4676
> URL: https://issues.apache.org/jira/browse/KAFKA-4676
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Vishal Shukla
>Priority: Critical
>  Labels: consumer, reliability
> Attachments: stuck-topic-thread-dump.log
>
>
> We recently upgraded to Kafka 0.10.1.0. We are frequently facing issue that 
> Kafka consumers get stuck suddenly for some partitions.
> Attached thread dump.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Dong Lin]

Yeah, your solution of adding new APIs certainly works and I don't think
that is an issue. On the other hand I don't think it is an issue to add a
new checkpoint file as well since we already have multiple checkpoint
files. The benefit of the new approach you mentioned is probably not an
issue in the current approach since high watermark and low watermark works
completely independently. Since there is no strong reason to choose either
of them, I am inclined to choose the one that makes less format change and
simpler in the Java API. The current approach seems better w.r.t this minor
reason.

If you are strong that we should use the new approach, I can do that as
well. Please let me know if you think so, and I will need to ask
Jun/Joel/Becket to vote on this again since this changes the interface of
the KIP.

On Sun, Jan 22, 2017 at 9:35 AM, Guozhang Wang  wrote:

> I think this is less of an issue: we can use the same patterns as in the
> request protocol, i.e.:
>
> write(Map[TP, Long]) // write the checkout point in v0 format
> write(Map[TP, Pair[Long, Long]]) // write the checkout point in v1 format
>
> CheckpointedOffsets read() // read the file relying on its version id
>
> class CheckpointedOffsets {
>
> Integer getVersion();
> Long getFirstOffset();
> Long getSecondOffset();   // would return NO_AVAILABLE with v0 format
> }
>
>
> As I think of it, another benefit is that we wont have a partition that
> only have one of the watermarks in case of a failure in between writing two
> files.
>
> Guozhang
>
> On Sun, Jan 22, 2017 at 12:03 AM, Dong Lin  wrote:
>
> > Hey Guozhang,
> >
> > Thanks for the review:) Yes it is possible to combine them. Both solution
> > will have the same performance. But I think the current solution will
> give
> > us simpler Java class design. Note that we will have to change Java API
> > (e.g. read() and write()) of OffsetCheckpoint class in order to provide a
> > map from TopicPartition to a pair of integers when we write to checkpoint
> > file. This makes this class less generic since this API is not used by
> log
> > recovery checkpoint and log cleaner checkpoint which are also using
> > OffsetCheckpoint class.
> >
> > Dong
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Jan 21, 2017 at 12:28 PM, Guozhang Wang 
> > wrote:
> >
> > > Hi Dong,
> > >
> > > Sorry for being late on reviewing this KIP. It LGTM overall, but I'm
> > > wondering if we can save adding the "replication-low-watermark-
> > checkpoint"
> > > file by just bumping up the version number of "replication-offset-
> > > checkpoint"
> > > to let it have two values for each partition, i.e.:
> > >
> > > 1  // version number
> > > [number of partitions]
> > > [topic name] [partition id] [lwm] [hwm]
> > >
> > >
> > > This will affects the upgrade path a bit, but I think not by large, and
> > all
> > > other logic will not be affected.
> > >
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Wed, Jan 18, 2017 at 6:12 PM, Dong Lin  wrote:
> > >
> > > > Thanks to everyone who voted and provided feedback!
> > > >
> > > > This KIP is now adopted with 3 binding +1s (Jun, Joel, Becket) and 2
> > > > non-binding +1s (Radai, Mayuresh).
> > > >
> > > > Thanks,
> > > > Dong
> > > >
> > > > On Wed, Jan 18, 2017 at 6:05 PM, Jun Rao  wrote:
> > > >
> > > > > Hi, Dong,
> > > > >
> > > > > Thanks for the update. +1
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jan 18, 2017 at 1:44 PM, Dong Lin 
> > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > After some more thinking, I agree with you that it is better to
> > > simply
> > > > > > throw OffsetOutOfRangeException and not update low_watermark if
> > > > > > offsetToPurge is larger than high_watermark.
> > > > > >
> > > > > > My use-case of allowing low_watermark > high_watermark in 2(b) is
> > to
> > > > > allow
> > > > > > user to purge all the data in the log even if that data is not
> > fully
> > > > > > replicated to followers. An offset higher than high_watermark may
> > be
> > > > > > returned to user either through producer's RecordMetadata, or
> > through
> > > > > > ListOffsetResponse if from_consumer option is false. However,
> this
> > > may
> > > > > > cause problem in case of unclean leader election or when consumer
> > > seeks
> > > > > to
> > > > > > the largest offset of the partition. It will complicate this KIP
> if
> > > we
> > > > > were
> > > > > > to address these two problems.
> > > > > >
> > > > > > At this moment I prefer to keep this KIP simple by requiring
> > > > > low_watermark
> > > > > > <= high_watermark. The caveat is that if user does want to purge
> > > *all*
> > > > > the
> > > > > > data that is already produced, then he needs to stop all
> producers
> > > that
> > > > > are
> > > > > > producing into this topic, wait long enough for all followers to
> > > catch
> > > > > up,
> > > > > > and then purge data 

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833708#comment-15833708
 ] 

Jeyhun Karimov commented on KAFKA-4144:
---

[~elevy] I am not sure I got your point. Please correct me if I am wrong in my 
assumptions. Lets look at different possibilities:

1. We create separate {{builder.stream/table(topicName)}} instances in 
application. So we clearly separate {{KTable}}/{{KStream}}s, one per topic. In 
this case, we can develop a solution described above. So, create different 
{{TimestampExtractor}} and as a result, assign one {{timeStampExtractor} per 
topic.

2. If we create {{builder.stream/table(topicName1, topicName2 ...)}} this can 
be challenging, as there is no clear separation of topics within stream 
application. 

I think you pointed out the second case. As a solution, we can provide 
{{Map}} to topology, indicating which 
{{TimestampExtractor}} is related to which topic. As a result, if the topics 
can change in runtime, the stream application can handle it.

Please correct me if I am wrong in my assumptions.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833686#comment-15833686
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 58 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 8:32:59 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 19:41:35 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 19:41:35 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd3-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd3-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id 0FCB6400C6
for ; Sun, 22 Jan 2017 17:41:33 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd3-us-west.apache.org (ASF Mail Server at 
spamd3-us-west.apache.org) with ESMTP id CF8B118C334
for ; Sun, 22 Jan 2017 16:41:33 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, 
port 10024)
with ESMTP id 0jfQf1ZTR2SX for ;
Sun, 22 Jan 2017 16:41:32 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with 
ESMTP id 1E1815FC9D
for ; Sun, 22 Jan 2017 16:41:30 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 3F1B1E031F
for ; Sun, 22 Jan 2017 16:41:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id 99B592528D
for ; Sun, 22 Jan 2017 16:41:26 + (UTC)
Date: Sun, 22 Jan 2017 16:41:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: 0FCB6400C6.A79EA
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485708094.88981@I+Vmr+QdtYGPQMW8negScQ
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833687#comment-15833687
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 58 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 8:32:59 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 19:41:34 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 19:41:34 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd3-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd3-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id CAF86400C3
for ; Sun, 22 Jan 2017 17:41:33 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd3-us-west.apache.org (ASF Mail Server at 
spamd3-us-west.apache.org) with ESMTP id 7BBF61806F7
for ; Sun, 22 Jan 2017 16:41:33 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, 
port 10024)
with ESMTP id quuaPayVMB7c for ;
Sun, 22 Jan 2017 16:41:31 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with 
ESMTP id 8E79B60DE7
for ; Sun, 22 Jan 2017 16:41:30 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id C4297E040F
for ; Sun, 22 Jan 2017 16:41:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id CEBF2252A1
for ; Sun, 22 Jan 2017 16:41:26 + (UTC)
Date: Sun, 22 Jan 2017 16:41:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: CAF86400C3.A7AA0
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485708094.24373@VgpCJ/pJvQR0MMzQjQbSiQ
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-4144) Allow per stream/table timestamp extractor

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833649#comment-15833649
 ] 

Elias Levy commented on KAFKA-4144:
---

That's fine.  I am less concerned about the implementation details than about 
the developer's API experience.  E.g. having {{TopologyBuilder.addSource}}, 
{{KStreamBuilder.source}}, and {{KStreamBuilder.table}} take a 
{{TimestampExtractor}}.

There is another problem with {{TimestampExtractor}}.  It is instantiated 
within StreamTask and the instances cannot be configured.  That means that the 
topics that {{TimestampExtractor}} supports are hardcoded into it. That makes 
it impossible to have a Stream application that uses {{TimestampExtractor}} 
were the input topics can be configured at execution time.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3628) Native Schema Registry in Kafka

[Sushant Sood (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833640#comment-15833640
 ] 

Sushant Sood commented on KAFKA-3628:
-

Hi [~sriharsha] can you please share the updates on this Feature . 

> Native Schema Registry in Kafka
> ---
>
> Key: KAFKA-3628
> URL: https://issues.apache.org/jira/browse/KAFKA-3628
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>
> Instead of having external schema service. We can use topic config store the 
> schema. I'll write detailed KIP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Guozhang Wang]

I think it also affects KIP-113 design but just leave it as a single
comment here.

On Sun, Jan 22, 2017 at 10:50 AM, Guozhang Wang  wrote:

> Hello Dong,
>
> Thanks for the very well written KIP. I had a general thought on the ZK
> path management, wondering if the following alternative would work:
>
> 1. Bump up versions in "brokers/topics/[topic]" and "
> /brokers/topics/[topic]/partitions/[partitionId]/state" to 2, in which
> the replica id is no longer an int but a string.
>
> 2. Bump up versions in "/brokers/ids/[brokerId]" to add another field:
>
> { "fields":
> [ {"name": "version", "type": "int", "doc": "version id"},
>   {"name": "host", "type": "string", "doc": "ip address or host name
> of the broker"},
>   {"name": "port", "type": "int", "doc": "port of the broker"},
>   {"name": "jmx_port", "type": "int", "doc": "port for jmx"}
>   {"name": "log_dirs",
>"type": {"type": "array",
> "items": "int",
> "doc": "an array of the id of the log dirs in broker"}
>   },
> ]
> }
>
> 3. The replica id can now either be an string-typed integer indicating
> that all partitions on the broker still treated as failed or not as a
> whole, i.e. no support needed for JBOD; or be a string typed
> "[brokerID]-[dirID]", in which brokers / controllers can still parse to
> determine which broker is hosting this replica: in this case the management
> of replicas is finer grained, no longer at the broker level (i.e. if broker
> dies all replicas go offline) but broker-dir level.
>
> 4. When broker had one of the dir failed, it can modify its "
> /brokers/ids/[brokerId]" registry and remove the dir id, controller
> already listening on this path can then be notified and run the replica
> assignment accordingly where replica id is computed as above.
>
>
> By doing this controller can also naturally reassign replicas between dirs
> within the same broker.
>
>
> Guozhang
>
>
> On Thu, Jan 12, 2017 at 6:25 PM, Ismael Juma  wrote:
>
>> Thanks for the KIP. Just wanted to quickly say that it's great to see
>> proposals for improving JBOD (KIP-113 too). More feedback soon, hopefully.
>>
>> Ismael
>>
>> On Thu, Jan 12, 2017 at 6:46 PM, Dong Lin  wrote:
>>
>> > Hi all,
>> >
>> > We created KIP-112: Handle disk failure for JBOD. Please find the KIP
>> wiki
>> > in the link https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 112%3A+Handle+disk+failure+for+JBOD.
>> >
>> > This KIP is related to KIP-113
>> > > > 113%3A+Support+replicas+movement+between+log+directories>:
>> > Support replicas movement between log directories. They are needed in
>> order
>> > to support JBOD in Kafka. Please help review the KIP. You feedback is
>> > appreciated!
>> >
>> > Thanks,
>> > Dong
>> >
>>
>
>
>
> --
> -- Guozhang
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[Guozhang Wang]

Hello Dong,

Thanks for the very well written KIP. I had a general thought on the ZK
path management, wondering if the following alternative would work:

1. Bump up versions in "brokers/topics/[topic]" and
"/brokers/topics/[topic]/partitions/[partitionId]/state"
to 2, in which the replica id is no longer an int but a string.

2. Bump up versions in "/brokers/ids/[brokerId]" to add another field:

{ "fields":
[ {"name": "version", "type": "int", "doc": "version id"},
  {"name": "host", "type": "string", "doc": "ip address or host name of
the broker"},
  {"name": "port", "type": "int", "doc": "port of the broker"},
  {"name": "jmx_port", "type": "int", "doc": "port for jmx"}
  {"name": "log_dirs",
   "type": {"type": "array",
"items": "int",
"doc": "an array of the id of the log dirs in broker"}
  },
]
}

3. The replica id can now either be an string-typed integer indicating that
all partitions on the broker still treated as failed or not as a whole,
i.e. no support needed for JBOD; or be a string typed "[brokerID]-[dirID]",
in which brokers / controllers can still parse to determine which broker is
hosting this replica: in this case the management of replicas is finer
grained, no longer at the broker level (i.e. if broker dies all replicas go
offline) but broker-dir level.

4. When broker had one of the dir failed, it can modify its "
/brokers/ids/[brokerId]" registry and remove the dir id, controller already
listening on this path can then be notified and run the replica assignment
accordingly where replica id is computed as above.


By doing this controller can also naturally reassign replicas between dirs
within the same broker.


Guozhang


On Thu, Jan 12, 2017 at 6:25 PM, Ismael Juma  wrote:

> Thanks for the KIP. Just wanted to quickly say that it's great to see
> proposals for improving JBOD (KIP-113 too). More feedback soon, hopefully.
>
> Ismael
>
> On Thu, Jan 12, 2017 at 6:46 PM, Dong Lin  wrote:
>
> > Hi all,
> >
> > We created KIP-112: Handle disk failure for JBOD. Please find the KIP
> wiki
> > in the link https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 112%3A+Handle+disk+failure+for+JBOD.
> >
> > This KIP is related to KIP-113
> >  > 113%3A+Support+replicas+movement+between+log+directories>:
> > Support replicas movement between log directories. They are needed in
> order
> > to support JBOD in Kafka. Please help review the KIP. You feedback is
> > appreciated!
> >
> > Thanks,
> > Dong
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Guozhang Wang]

I think this is less of an issue: we can use the same patterns as in the
request protocol, i.e.:

write(Map[TP, Long]) // write the checkout point in v0 format
write(Map[TP, Pair[Long, Long]]) // write the checkout point in v1 format

CheckpointedOffsets read() // read the file relying on its version id

class CheckpointedOffsets {

Integer getVersion();
Long getFirstOffset();
Long getSecondOffset();   // would return NO_AVAILABLE with v0 format
}


As I think of it, another benefit is that we wont have a partition that
only have one of the watermarks in case of a failure in between writing two
files.

Guozhang

On Sun, Jan 22, 2017 at 12:03 AM, Dong Lin  wrote:

> Hey Guozhang,
>
> Thanks for the review:) Yes it is possible to combine them. Both solution
> will have the same performance. But I think the current solution will give
> us simpler Java class design. Note that we will have to change Java API
> (e.g. read() and write()) of OffsetCheckpoint class in order to provide a
> map from TopicPartition to a pair of integers when we write to checkpoint
> file. This makes this class less generic since this API is not used by log
> recovery checkpoint and log cleaner checkpoint which are also using
> OffsetCheckpoint class.
>
> Dong
>
>
>
>
>
>
>
> On Sat, Jan 21, 2017 at 12:28 PM, Guozhang Wang 
> wrote:
>
> > Hi Dong,
> >
> > Sorry for being late on reviewing this KIP. It LGTM overall, but I'm
> > wondering if we can save adding the "replication-low-watermark-
> checkpoint"
> > file by just bumping up the version number of "replication-offset-
> > checkpoint"
> > to let it have two values for each partition, i.e.:
> >
> > 1  // version number
> > [number of partitions]
> > [topic name] [partition id] [lwm] [hwm]
> >
> >
> > This will affects the upgrade path a bit, but I think not by large, and
> all
> > other logic will not be affected.
> >
> >
> > Guozhang
> >
> >
> >
> > On Wed, Jan 18, 2017 at 6:12 PM, Dong Lin  wrote:
> >
> > > Thanks to everyone who voted and provided feedback!
> > >
> > > This KIP is now adopted with 3 binding +1s (Jun, Joel, Becket) and 2
> > > non-binding +1s (Radai, Mayuresh).
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Wed, Jan 18, 2017 at 6:05 PM, Jun Rao  wrote:
> > >
> > > > Hi, Dong,
> > > >
> > > > Thanks for the update. +1
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jan 18, 2017 at 1:44 PM, Dong Lin 
> wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > After some more thinking, I agree with you that it is better to
> > simply
> > > > > throw OffsetOutOfRangeException and not update low_watermark if
> > > > > offsetToPurge is larger than high_watermark.
> > > > >
> > > > > My use-case of allowing low_watermark > high_watermark in 2(b) is
> to
> > > > allow
> > > > > user to purge all the data in the log even if that data is not
> fully
> > > > > replicated to followers. An offset higher than high_watermark may
> be
> > > > > returned to user either through producer's RecordMetadata, or
> through
> > > > > ListOffsetResponse if from_consumer option is false. However, this
> > may
> > > > > cause problem in case of unclean leader election or when consumer
> > seeks
> > > > to
> > > > > the largest offset of the partition. It will complicate this KIP if
> > we
> > > > were
> > > > > to address these two problems.
> > > > >
> > > > > At this moment I prefer to keep this KIP simple by requiring
> > > > low_watermark
> > > > > <= high_watermark. The caveat is that if user does want to purge
> > *all*
> > > > the
> > > > > data that is already produced, then he needs to stop all producers
> > that
> > > > are
> > > > > producing into this topic, wait long enough for all followers to
> > catch
> > > > up,
> > > > > and then purge data using the latest offset of this partition, i.e.
> > > > > high_watermark. We can revisit this if some strong use-case comes
> up
> > in
> > > > the
> > > > > future.
> > > > >
> > > > > I also updated the KIP to allow user to use offset -1L to indicate
> > > > > high_watermark in the PurgeRequest. In the future we can allow
> users
> > to
> > > > use
> > > > > offset -2L to indicate that they want to purge all data up to
> > > > logEndOffset.
> > > > >
> > > > > Thanks!
> > > > > Dong
> > > > >
> > > > >
> > > > > On Wed, Jan 18, 2017 at 10:37 AM, Jun Rao 
> wrote:
> > > > >
> > > > > > Hi, Dong,
> > > > > >
> > > > > > For 2(b), it seems a bit weird to allow highWatermark to be
> smaller
> > > > than
> > > > > > lowWatermark. Also, from the consumer's perspective, messages are
> > > > > available
> > > > > > only up to highWatermark. What if we simply throw
> > > > > OffsetOutOfRangeException
> > > > > > if offsetToPurge is larger than highWatermark?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, Jan 17, 2017 at 9:54 PM, Dong Lin 
> > > wrote:
> > > 

[jira] [Commented] (KAFKA-4304) Extend Interactive Queries for return latest update timestamp per key

[Jeyhun Karimov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833607#comment-15833607
 ] 

Jeyhun Karimov commented on KAFKA-4304:
---

[~mjsax] Can we generalize this feature to {{top-k}} queries?

> Extend Interactive Queries for return latest update timestamp per key
> -
>
> Key: KAFKA-4304
> URL: https://issues.apache.org/jira/browse/KAFKA-4304
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Priority: Minor
>  Labels: newbie++
>
> Currently, when querying state store, it is not clear when the key was 
> updated last. The ides of this JIRA is to make the latest update timestamp 
> for each key-value-pair of the state store accessible.
> For example, this might be useful to
>  * check if a value was update but did not changed (just compare the update 
> TS)
>  * if you want to consider only recently updated keys



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833597#comment-15833597
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 51 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-02.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 4:29:42 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-02.inn.local
 (10.64.37.98) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 15:31:35 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 15:31:35 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd3-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd3-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id DF86B400C6
for ; Sun, 22 Jan 2017 13:31:34 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd3-us-west.apache.org (ASF Mail Server at 
spamd3-us-west.apache.org) with ESMTP id 8D7F71806F7
for ; Sun, 22 Jan 2017 12:31:34 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, 
port 10024)
with ESMTP id gAFEOAt8mxtY for ;
Sun, 22 Jan 2017 12:31:32 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with 
ESMTP id 935CC5F4AC
for ; Sun, 22 Jan 2017 12:31:31 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 535EEE0366
for ; Sun, 22 Jan 2017 12:31:27 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id AE2812528E
for ; Sun, 22 Jan 2017 12:31:26 + (UTC)
Date: Sun, 22 Jan 2017 12:31:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: DF86B400C6.A9951
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485693095.59716@y4RZfH4sHX1Lm3/FWf7QEw
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-02.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-02.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833596#comment-15833596
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 51 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-02.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 4:29:42 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-02.inn.local
 (10.64.37.98) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 15:31:35 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 15:31:35 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd2-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd2-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id 8BB67400C3
for ; Sun, 22 Jan 2017 13:31:34 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd2-us-west.apache.org (ASF Mail Server at 
spamd2-us-west.apache.org) with ESMTP id EA3F31A031A
for ; Sun, 22 Jan 2017 12:31:33 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 
10024)
with ESMTP id 5lM0TY03erlc for ;
Sun, 22 Jan 2017 12:31:32 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with 
ESMTP id 1DC6A5FCD2
for ; Sun, 22 Jan 2017 12:31:31 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id E88B9E0410
for ; Sun, 22 Jan 2017 12:31:27 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id EDF8D252A0
for ; Sun, 22 Jan 2017 12:31:26 + (UTC)
Date: Sun, 22 Jan 2017 12:31:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: 8BB67400C3.A8C37
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485693095.59482@JfEAIAbQf1GSSUpV86wPsQ
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-02.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-02.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Assigned] (KAFKA-4613) Treat null-key records the same way for joins and aggreations

[Jeyhun Karimov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeyhun Karimov reassigned KAFKA-4613:
-

Assignee: Jeyhun Karimov

> Treat null-key records the same way for joins and aggreations
> -
>
> Key: KAFKA-4613
> URL: https://issues.apache.org/jira/browse/KAFKA-4613
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
>
> Currently, on aggregation records with null-key get dropped while for joins 
> we raise an exception.
> We might want to drop in both cases of raise an exception in both cases to be 
> consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4613) Treat null-key records the same way for joins and aggreations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833586#comment-15833586
 ] 

ASF GitHub Bot commented on KAFKA-4613:
---

GitHub user jeyhunkarimov opened a pull request:

https://github.com/apache/kafka/pull/2420

KAFKA-4613: Treat null-key records the same way for joins and aggreations



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeyhunkarimov/kafka KAFKA-4613

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2420


commit 70c23d0d77d9a3e863d6a16cce632fe23ced1a6c
Author: Jeyhun Karimov 
Date:   2017-01-22T16:35:54Z

KAFKA-4613: Treat null-key records the same way for joins and aggreations




> Treat null-key records the same way for joins and aggreations
> -
>
> Key: KAFKA-4613
> URL: https://issues.apache.org/jira/browse/KAFKA-4613
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>
> Currently, on aggregation records with null-key get dropped while for joins 
> we raise an exception.
> We might want to drop in both cases of raise an exception in both cases to be 
> consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2420: KAFKA-4613: Treat null-key records the same way fo...

[jeyhunkarimov]

GitHub user jeyhunkarimov opened a pull request:

https://github.com/apache/kafka/pull/2420

KAFKA-4613: Treat null-key records the same way for joins and aggreations



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeyhunkarimov/kafka KAFKA-4613

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2420


commit 70c23d0d77d9a3e863d6a16cce632fe23ced1a6c
Author: Jeyhun Karimov 
Date:   2017-01-22T16:35:54Z

KAFKA-4613: Treat null-key records the same way for joins and aggreations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833131#comment-15833131
 ] 

Jeyhun Karimov edited comment on KAFKA-4144 at 1/22/17 3:06 PM:


[~elevy] After checking the code, I think we should add extra  
{{timestampextractor}} field to {{SourceNode}} class. In this case there will 
be one {{timeStampExtractor}} per topic partition. As a result, in 
{{StreamTask}} constructor, we can get access to particular {{SourceNode}}'s 
{{timeStampExtractor}}. Please let me know your comments on this approach. 


was (Author: jeyhunkarimov):
[~elevy] After checking the code, I think we should add extra  {{ 
timestampextractor }} field to {{ SourceNode }} class. In this case there will 
be one {{ timeStampExtractor }} per topic partition. As a result, in {{ 
StreamTask}} constructor, we can get access to particular {{ SourceNode }}'s {{ 
timeStampExtractor }}. Please let me know your comments on this approach. 

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4676) Kafka consumers gets stuck for some partitions

[Bhavesh Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833429#comment-15833429
 ] 

Bhavesh Shah edited comment on KAFKA-4676 at 1/22/17 3:04 PM:
--

Hi [~hachikuji]/[~ijuma],

Even after upgrading kafka to 0.10.1.1 we encountered kafka topics were getting 
stuck again. [All below logs are after upgrading to 0.10.1.1]

Logs from consumer application around the time topics were stuck. 

{code}
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset 
commit failed for group dummy-consumer-group: Commit cannot be completed since 
the group has already rebalanced and assigned the partitions to another member. 
This maeans that the time between subsequent calls to poll() was longer than 
the configured max.poll.interval.ms, which typically implies that the poll loop 
is spending too much time message processing. You can address this either by 
increasing the session timeout or by reducing the maximum size of batches 
returned in poll() with max.poll.records.
{code}

Apart from below, Couldn't find anything significant from broker/controller 
logs around the time topic was stuck

{code}
[2017-01-22 03:30:05,383] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 80 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:07,949] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 81 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:07,955] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 81 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:07,963] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 81 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:10,972] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 82 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:10,973] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 82 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:11,373] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 82 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:13,986] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 83 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:13,987] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 83 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:14,744] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 83 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:15,889] INFO [Group Metadata Manager on Broker 0]: Removed 0 
expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-01-22 03:30:16,992] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 84 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:16,993] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 84 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:18,131] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 84 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:19,998] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 85 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:19,999] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 85 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:20,509] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 85 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:23,004] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 86 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:23,006] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 86 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:23,079] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 86 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:25,379] INFO [GroupCoordinator 0]: Group dummy-consumer-group 
with generation 87 is now empty (kafka.coordinator.GroupCoordinator)
{code}


We also made below consumer config values configurable. Please let us know if 
they are in-appropriate/unusual.

{code}
session.timeout.ms=30
max.poll.interval.ms=30
max.poll.records=100
request.timeout.ms=305
{code}

Keeping eyes on broker logs to discover something helpful with regards to 
reported issue.


was (Author: bhavesh.shah):
Hi 

[jira] [Comment Edited] (KAFKA-4144) Allow per stream/table timestamp extractor

[Jeyhun Karimov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833131#comment-15833131
 ] 

Jeyhun Karimov edited comment on KAFKA-4144 at 1/22/17 3:05 PM:


[~elevy] After checking the code, I think we should add extra  {{ 
timestampextractor }} field to {{ SourceNode }} class. In this case there will 
be one {{ timeStampExtractor }} per topic partition. As a result, in {{ 
StreamTask}} constructor, we can get access to particular {{ SourceNode }}'s {{ 
timeStampExtractor }}. Please let me know your comments on this approach. 


was (Author: jeyhunkarimov):
[~elevy] Shouldn't it be per partitioned stream/table? So, we can define 
'TimestampExtractor' per application, per stream/table and per partitioned 
stream/table ('KGoupedStream', 'KGoupedTable') and the latter would override 
the former.

> Allow per stream/table timestamp extractor
> --
>
> Key: KAFKA-4144
> URL: https://issues.apache.org/jira/browse/KAFKA-4144
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Jeyhun Karimov
>  Labels: api
>
> At the moment the timestamp extractor is configured via a StreamConfig value 
> to KafkaStreams.  That means you can only have a single timestamp extractor 
> per app, even though you may be joining multiple streams/tables that require 
> different timestamp extraction methods.
> You should be able to specify a timestamp extractor via 
> KStreamBuilder.stream/table, just like you can specify key and value serdes 
> that override the StreamConfig defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2000) Delete consumer offsets from kafka once the topic is deleted

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2000:
---
Reviewer: Jason Gustafson  (was: Joel Koshy)

> Delete consumer offsets from kafka once the topic is deleted
> 
>
> Key: KAFKA-2000
> URL: https://issues.apache.org/jira/browse/KAFKA-2000
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-2000_2015-05-03_10:39:11.patch, KAFKA-2000.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833491#comment-15833491
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 55 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-02.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 12:19:41 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-02.inn.local
 (10.64.37.98) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 11:24:52 +0300
Received: from lc-asp-02.inn.ru (10.64.37.104) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 11:24:52 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd2-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd2-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id AAD66400C6
for ; Sun, 22 Jan 2017 09:24:51 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd2-us-west.apache.org (ASF Mail Server at 
spamd2-us-west.apache.org) with ESMTP id 496481A0049
for ; Sun, 22 Jan 2017 08:24:51 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 
10024)
with ESMTP id 8Xtom6xPm6iN for ;
Sun, 22 Jan 2017 08:24:49 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with 
ESMTP id 5688D5F367
for ; Sun, 22 Jan 2017 08:24:49 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 6F3C6E0416
for ; Sun, 22 Jan 2017 08:24:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id 34F122529F
for ; Sun, 22 Jan 2017 08:24:27 + (UTC)
Date: Sun, 22 Jan 2017 08:24:27 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: AAD66400C6.A7E77
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485678292.66566@EMpz+EGjnPSFw01eT33gpA
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-02.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-02.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833492#comment-15833492
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 54 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-02.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 12:19:41 PM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-02.inn.local
 (10.64.37.98) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 11:24:33 +0300
Received: from lc-asp-02.inn.ru (10.64.37.104) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 11:24:33 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd3-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd3-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id 460A1400C6
for ; Sun, 22 Jan 2017 09:24:33 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd3-us-west.apache.org (ASF Mail Server at 
spamd3-us-west.apache.org) with ESMTP id 50523181BC0
for ; Sun, 22 Jan 2017 08:24:32 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, 
port 10024)
with ESMTP id uPl-G8z3QpFR for ;
Sun, 22 Jan 2017 08:24:30 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with 
ESMTP id B5BF65FB6A
for ; Sun, 22 Jan 2017 08:24:29 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 891AFE036B
for ; Sun, 22 Jan 2017 08:24:27 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id AA6CC2528D
for ; Sun, 22 Jan 2017 08:24:26 + (UTC)
Date: Sun, 22 Jan 2017 08:24:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: 460A1400C6.A5784
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485678273.65541@h/zNqAT94w6h8y7PE7IQAA
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-02.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-02.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-4676) Kafka consumers gets stuck for some partitions

[Bhavesh Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833429#comment-15833429
 ] 

Bhavesh Shah commented on KAFKA-4676:
-

Hi [~hachikuji]/[~ijuma],

Event after upgrading kafka to 0.10.1.1 we encountered kafka topics were 
getting stuck again. [All below logs are after upgrading to 0.10.1.1]

Logs from consumer application around the time topics were stuck. 

{code}
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset 
commit failed for group dummy-consumer-group: Commit cannot be completed since 
the group has already rebalanced and assigned the partitions to another member. 
This maeans that the time between subsequent calls to poll() was longer than 
the configured max.poll.interval.ms, which typically implies that the poll loop 
is spending too much time message processing. You can address this either by 
increasing the session timeout or by reducing the maximum size of batches 
returned in poll() with max.poll.records.
{code}

Apart from below, Couldn't find anything significant from broker/controller 
logs around the time topic was stuck

{code}
[2017-01-22 03:30:05,383] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 80 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:07,949] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 81 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:07,955] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 81 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:07,963] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 81 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:10,972] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 82 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:10,973] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 82 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:11,373] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 82 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:13,986] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 83 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:13,987] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 83 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:14,744] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 83 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:15,889] INFO [Group Metadata Manager on Broker 0]: Removed 0 
expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-01-22 03:30:16,992] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 84 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:16,993] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 84 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:18,131] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 84 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:19,998] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 85 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:19,999] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 85 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:20,509] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 85 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:23,004] INFO [GroupCoordinator 0]: Stabilized group 
dummy-consumer-group generation 86 (kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:23,006] INFO [GroupCoordinator 0]: Assignment received from 
leader for group dummy-consumer-group for generation 86 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:23,079] INFO [GroupCoordinator 0]: Preparing to restabilize 
group dummy-consumer-group with old generation 86 
(kafka.coordinator.GroupCoordinator)
[2017-01-22 03:30:25,379] INFO [GroupCoordinator 0]: Group dummy-consumer-group 
with generation 87 is now empty (kafka.coordinator.GroupCoordinator)
{code}


We also made below consumer config values configurable. Please let us know if 
they are in-appropriate/unusual.

{code}
session.timeout.ms=30
max.poll.interval.ms=30
max.poll.records=100
request.timeout.ms=305
{code}

Keeping eyes on broker logs to discover something helpful with regards to 
reported issue.

> Kafka consumers gets stuck for some partitions
> 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833356#comment-15833356
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 51 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 8:12:55 AM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 07:14:33 +0300
Received: from lc-asp-02.inn.ru (10.64.37.104) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 07:14:33 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd1-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd1-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id DFB6C400C6
for ; Sun, 22 Jan 2017 05:14:32 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd1-us-west.apache.org (ASF Mail Server at 
spamd1-us-west.apache.org) with ESMTP id B5259C031C
for ; Sun, 22 Jan 2017 04:14:31 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-us.apache.org ([10.40.0.8])
by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 
10024)
with ESMTP id UKQsIfs1X0-9 for ;
Sun, 22 Jan 2017 04:14:30 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with 
ESMTP id 208445FE1E
for ; Sun, 22 Jan 2017 04:14:30 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 01136E02AA
for ; Sun, 22 Jan 2017 04:14:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id C01B62528D
for ; Sun, 22 Jan 2017 04:14:26 + (UTC)
Date: Sun, 22 Jan 2017 04:14:26 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: DFB6C400C6.A8AE5
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485663273.25183@fiEDDMEunYDnFCRbsn/HAg
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

[jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

[postmas...@inn.ru (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833355#comment-15833355
 ] 

postmas...@inn.ru commented on KAFKA-1207:
--

Delivery is delayed to these recipients or groups:

e...@inn.ru

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache Mesos

This message hasn't been delivered yet. Delivery will continue to be attempted.

The server will keep trying to deliver this message for the next 1 days, 19 
hours and 51 minutes. You'll be notified if the message can't be delivered by 
that time.







Diagnostic information for administrators:

Generating server: lc-exch-04.inn.local
Receiving server: inn.ru (109.105.153.25)

e...@inn.ru
Server at inn.ru (109.105.153.25) returned '400 4.4.7 Message delayed'
1/22/2017 8:12:55 AM - Server at inn.ru (109.105.153.25) returned '441 4.4.1 
Error communicating with target host: "Failed to connect. Winsock error code: 
10060, Win32 error code: 10060." Last endpoint attempted was 109.105.153.25:25'

Original message headers:

Received: from lc-exch-04.inn.local (10.64.37.99) by lc-exch-04.inn.local
 (10.64.37.99) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32; Sun, 22
 Jan 2017 07:14:35 +0300
Received: from lc-asp-02.inn.ru (10.64.37.105) by lc-exch-04.inn.local
 (10.64.37.100) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.669.32 via
 Frontend Transport; Sun, 22 Jan 2017 07:14:35 +0300
Received-SPF: None (no SPF record) identity=mailfrom; client-ip=209.188.14.142; 
helo=spamd2-us-west.apache.org; envelope-from=j...@apache.org; 
receiver=e...@inn.ru
X-Envelope-From: 
Received: from spamd2-us-west.apache.org (pnap-us-west-generic-nat.apache.org 
[209.188.14.142])
by lc-asp-02.inn.ru (Postfix) with ESMTP id 5DBE5400C3
for ; Sun, 22 Jan 2017 05:14:35 +0100 (CET)
Received: from localhost (localhost [127.0.0.1])
by spamd2-us-west.apache.org (ASF Mail Server at 
spamd2-us-west.apache.org) with ESMTP id D6B1F1A004B
for ; Sun, 22 Jan 2017 04:14:34 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31
tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999]
autolearn=disabled
Received: from mx1-lw-eu.apache.org ([10.40.0.8])
by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 
10024)
with ESMTP id bV3fvu-ezll6 for ;
Sun, 22 Jan 2017 04:14:33 + (UTC)
Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org 
[209.188.14.139])
by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with 
ESMTP id 727275FCF8
for ; Sun, 22 Jan 2017 04:14:31 + (UTC)
Received: from jira-lw-us.apache.org (unknown [207.244.88.139])
by mailrelay1-us-west.apache.org (ASF Mail Server at 
mailrelay1-us-west.apache.org) with ESMTP id 89E3AE0312
for ; Sun, 22 Jan 2017 04:14:28 + (UTC)
Received: from jira-lw-us.apache.org (localhost [127.0.0.1])
by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) 
with ESMTP id 259C7252A1
for ; Sun, 22 Jan 2017 04:14:27 + (UTC)
Date: Sun, 22 Jan 2017 04:14:27 +
From: "postmas...@inn.ru (JIRA)" 
To: 
Message-ID: 
In-Reply-To: 
References:  

Subject: [jira] [Commented] (KAFKA-1207) Launch Kafka from within Apache
 Mesos
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-inn-MailScanner-ESVA-Information: Please contact  for more information
X-inn-MailScanner-ESVA-ID: 5DBE5400C3.A8247
X-inn-MailScanner-ESVA: Found to be clean
X-inn-MailScanner-ESVA-From: j...@apache.org
X-inn-MailScanner-ESVA-Watermark: 1485663275.81827@RlTB5vg/kKBS9YXEDAR/OA
Return-Path: j...@apache.org
X-OrganizationHeadersPreserved: lc-exch-04.inn.local
X-CrossPremisesHeadersFilteredByDsnGenerator: lc-exch-04.inn.local



> Launch Kafka from within Apache Mesos
> -
>
> Key: KAFKA-1207
> URL: https://issues.apache.org/jira/browse/KAFKA-1207
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: mesos
> Attachments: KAFKA-1207_2014-01-19_00:04:58.patch, 
> KAFKA-1207_2014-01-19_00:48:49.patch, KAFKA-1207.patch
>
>
> There are a few components to 

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

[Dong Lin]

Hey Guozhang,

Thanks for the review:) Yes it is possible to combine them. Both solution
will have the same performance. But I think the current solution will give
us simpler Java class design. Note that we will have to change Java API
(e.g. read() and write()) of OffsetCheckpoint class in order to provide a
map from TopicPartition to a pair of integers when we write to checkpoint
file. This makes this class less generic since this API is not used by log
recovery checkpoint and log cleaner checkpoint which are also using
OffsetCheckpoint class.

Dong







On Sat, Jan 21, 2017 at 12:28 PM, Guozhang Wang  wrote:

> Hi Dong,
>
> Sorry for being late on reviewing this KIP. It LGTM overall, but I'm
> wondering if we can save adding the "replication-low-watermark-checkpoint"
> file by just bumping up the version number of "replication-offset-
> checkpoint"
> to let it have two values for each partition, i.e.:
>
> 1  // version number
> [number of partitions]
> [topic name] [partition id] [lwm] [hwm]
>
>
> This will affects the upgrade path a bit, but I think not by large, and all
> other logic will not be affected.
>
>
> Guozhang
>
>
>
> On Wed, Jan 18, 2017 at 6:12 PM, Dong Lin  wrote:
>
> > Thanks to everyone who voted and provided feedback!
> >
> > This KIP is now adopted with 3 binding +1s (Jun, Joel, Becket) and 2
> > non-binding +1s (Radai, Mayuresh).
> >
> > Thanks,
> > Dong
> >
> > On Wed, Jan 18, 2017 at 6:05 PM, Jun Rao  wrote:
> >
> > > Hi, Dong,
> > >
> > > Thanks for the update. +1
> > >
> > > Jun
> > >
> > > On Wed, Jan 18, 2017 at 1:44 PM, Dong Lin  wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > After some more thinking, I agree with you that it is better to
> simply
> > > > throw OffsetOutOfRangeException and not update low_watermark if
> > > > offsetToPurge is larger than high_watermark.
> > > >
> > > > My use-case of allowing low_watermark > high_watermark in 2(b) is to
> > > allow
> > > > user to purge all the data in the log even if that data is not fully
> > > > replicated to followers. An offset higher than high_watermark may be
> > > > returned to user either through producer's RecordMetadata, or through
> > > > ListOffsetResponse if from_consumer option is false. However, this
> may
> > > > cause problem in case of unclean leader election or when consumer
> seeks
> > > to
> > > > the largest offset of the partition. It will complicate this KIP if
> we
> > > were
> > > > to address these two problems.
> > > >
> > > > At this moment I prefer to keep this KIP simple by requiring
> > > low_watermark
> > > > <= high_watermark. The caveat is that if user does want to purge
> *all*
> > > the
> > > > data that is already produced, then he needs to stop all producers
> that
> > > are
> > > > producing into this topic, wait long enough for all followers to
> catch
> > > up,
> > > > and then purge data using the latest offset of this partition, i.e.
> > > > high_watermark. We can revisit this if some strong use-case comes up
> in
> > > the
> > > > future.
> > > >
> > > > I also updated the KIP to allow user to use offset -1L to indicate
> > > > high_watermark in the PurgeRequest. In the future we can allow users
> to
> > > use
> > > > offset -2L to indicate that they want to purge all data up to
> > > logEndOffset.
> > > >
> > > > Thanks!
> > > > Dong
> > > >
> > > >
> > > > On Wed, Jan 18, 2017 at 10:37 AM, Jun Rao  wrote:
> > > >
> > > > > Hi, Dong,
> > > > >
> > > > > For 2(b), it seems a bit weird to allow highWatermark to be smaller
> > > than
> > > > > lowWatermark. Also, from the consumer's perspective, messages are
> > > > available
> > > > > only up to highWatermark. What if we simply throw
> > > > OffsetOutOfRangeException
> > > > > if offsetToPurge is larger than highWatermark?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Jan 17, 2017 at 9:54 PM, Dong Lin 
> > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thank you. Please see my answers below. The KIP is updated to
> > answer
> > > > > these
> > > > > > questions (see here
> > > > > >  > diffpagesbyversion.action
> > > ?
> > > > > > pageId=67636826=5=6>
> > > > > > ).
> > > > > >
> > > > > > 1. Yes, in this KIP we wait for all replicas. This is the same as
> > if
> > > > > > producer sends a messge with ack=all and isr=all_replicas. So it
> > > seems
> > > > > that
> > > > > > the comparison is OK?
> > > > > >
> > > > > > 2. Good point! I haven't thought about the case where the
> > > > user-specified
> > > > > > offset > logEndOffset. Please see answers below.
> > > > > >
> > > > > > a) If offsetToPurge < lowWatermark, the first condition
> > > > > > of DelayedOperationPurgatory will be satisfied immediately when
> > > broker
> > > > > > receives PurgeRequest. Broker will send PurgeResponse to admin
> > client
> > > > > >