Build failed in Jenkins: kafka-trunk-jdk8 #1066

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4387; KafkaConsumer should properly handle interrupts

--
[...truncated 8000 lines...]
kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.SslTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
STARTED

kafka.integration.SslTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.SslTopicMetadataTest > 

[jira] [Updated] (KAFKA-4470) Exception (java.lang.RuntimeException) encountered during startup: org.codehaus.jackson.JsonParseException:

[JianwenSun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JianwenSun updated KAFKA-4470:
--
Priority: Blocker  (was: Major)

> Exception (java.lang.RuntimeException) encountered during startup: 
> org.codehaus.jackson.JsonParseException: 
> 
>
> Key: KAFKA-4470
> URL: https://issues.apache.org/jira/browse/KAFKA-4470
> Project: Kafka
>  Issue Type: Bug
>Reporter: JianwenSun
>Priority: Blocker
>
> upgrade 2.1.13 to 3.9 error.
> we upgrade 1.2.4 thrift tables to 2.1.13 without any problems, but when 
> upgrade to higher 3.x it looks something wrong.
> any help?
> [root@bj-dev-infra-001 cassandra]# apache-cassandra-3.9/bin/cassandra -R -f
> CompilerOracle: dontinline 
> org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset 
> (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
> CompilerOracle: dontinline 
> org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset 
> (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
> CompilerOracle: dontinline 
> org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize 
> (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
> CompilerOracle: dontinline 
> org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
> CompilerOracle: dontinline 
> org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
> CompilerOracle: dontinline 
> org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
> CompilerOracle: dontinline 
> org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
> CompilerOracle: inline 
> org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody 
> (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
> CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
> CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds 
> (JJ)V
> CompilerOracle: inline 
> org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary 
> (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan 
> (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
> CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes 
> (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
> CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes 
> (JJIJ[J)V
> CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare 
> (Ljava/nio/ByteBuffer;[B)I
> CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare 
> ([BLjava/nio/ByteBuffer;)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned 
> (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
> (Ljava/lang/Object;JILjava/lang/Object;JI)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
> (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
> CompilerOracle: inline 
> org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
> (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
> CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt 
> (JI)[B
> INFO  06:05:28 Configuration location: 
> file:/usr/local/cassandra/apache-cassandra-3.9/conf/cassandra.yaml
> INFO  06:05:28 Node configuration:[allocate_tokens_for_keyspace=null; 
> authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; 
> auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; 
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; 
> broadcast_address=null; broadcast_rpc_address=null; 
> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; 
> cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
> cdc_raw_directory=/usr/local/cassandra/data/cdc_raw; 
> cdc_total_space_in_mb=null; client_encryption_options=; 
> cluster_name=TestCluster; column_index_cache_size_in_kb=2; 
> column_index_size_in_kb=64; commit_failure_policy=stop; 
> commitlog_compression=null; 
> commitlog_directory=/usr/local/cassandra/data/commitlog; 
> 

[jira] [Created] (KAFKA-4470) Exception (java.lang.RuntimeException) encountered during startup: org.codehaus.jackson.JsonParseException:

[JianwenSun (JIRA)]

JianwenSun created KAFKA-4470:
-

 Summary: Exception (java.lang.RuntimeException) encountered during 
startup: org.codehaus.jackson.JsonParseException: 
 Key: KAFKA-4470
 URL: https://issues.apache.org/jira/browse/KAFKA-4470
 Project: Kafka
  Issue Type: Bug
Reporter: JianwenSun


upgrade 2.1.13 to 3.9 error.

we upgrade 1.2.4 thrift tables to 2.1.13 without any problems, but when upgrade 
to higher 3.x it looks something wrong.

any help?


[root@bj-dev-infra-001 cassandra]# apache-cassandra-3.9/bin/cassandra -R -f
CompilerOracle: dontinline 
org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset 
(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline 
org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset 
(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline 
org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize 
(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline 
org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline 
org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline 
org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline 
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline 
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
CompilerOracle: dontinline 
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline 
org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline 
org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody 
(Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline 
org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary 
(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline 
org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan 
(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes 
(Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes 
(JJIJ[J)V
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare 
(Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare 
([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline 
org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned 
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline 
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
(Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline 
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline 
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo 
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt 
(JI)[B
INFO  06:05:28 Configuration location: 
file:/usr/local/cassandra/apache-cassandra-3.9/conf/cassandra.yaml
INFO  06:05:28 Node configuration:[allocate_tokens_for_keyspace=null; 
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; 
auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; 
batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; 
broadcast_address=null; broadcast_rpc_address=null; 
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; 
cdc_enabled=false; cdc_free_space_check_interval_ms=250; 
cdc_raw_directory=/usr/local/cassandra/data/cdc_raw; 
cdc_total_space_in_mb=null; client_encryption_options=; 
cluster_name=TestCluster; column_index_cache_size_in_kb=2; 
column_index_size_in_kb=64; commit_failure_policy=stop; 
commitlog_compression=null; 
commitlog_directory=/usr/local/cassandra/data/commitlog; 
commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; 
commitlog_segment_size_in_mb=32; commitlog_sync=periodic; 
commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=1; 
commitlog_total_space_in_mb=null; 
compaction_large_partition_warning_threshold_mb=100; 
compaction_throughput_mb_per_sec=0; concurrent_compactors=20; 
concurrent_counter_writes=32; concurrent_materialized_view_writes=32; 
concurrent_reads=32; 

[jira] [Commented] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707668#comment-15707668
 ] 

James Cheng commented on KAFKA-3042:


If KAFKA-4443 in fact solves this issue, then I'm ecstatically happy. This 
issue has been biting us for almost a year. Thanks [~lindong]!

> updateIsr should stop after failed several times due to zkVersion issue
> ---
>
> Key: KAFKA-3042
> URL: https://issues.apache.org/jira/browse/KAFKA-3042
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
> Environment: jdk 1.7
> centos 6.4
>Reporter: Jiahongchao
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.2.0
>
> Attachments: controller.log, server.log.2016-03-23-01, 
> state-change.log
>
>
> sometimes one broker may repeatly log
> "Cached zkVersion 54 not equal to that in zookeeper, skip updating ISR"
> I think this is because the broker consider itself as the leader in fact it's 
> a follower.
> So after several failed tries, it need to find out who is the leader



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 0.10.1.1 Plan

[Bernard Leach]

Hi Guozhang,

My suggestion was to not add kafka_2.12-0.10.1.0.tgz to downloads.html but to 
still run the build to generate the maven artefacts for 2.12 and still publish 
those to maven central.  That would allow projects with binary dependencies on 
kafka to obtain the required jars but hide away the tgz so as not to imply that 
it is suitable for production use.  Alternatively just do the regular release 
process and mark it as beta on the downloads page.  Either way you’ll get more 
exposure and testing of the 2.12 version which you won’t get via SNAPSHOTs from 
the trunk.,

cheers,
bern

> On 30 Nov 2016, at 16:52, Guozhang Wang  wrote:
> 
> @Ignacio Solis
> 
> The commit you mentioned was not intended for 0.10.1 but only for trunk
> (and there is a related KIP for this API change), but mistakenly gets
> leaked in and was already reverted.
> 
> @Bernard Leach
> 
> Could you elaborate on "instead simply publish the artifacts to maven
> central"? Currently the Kafka release is already through maven and we do
> not yet have kafka_2.12-0.10.1.0.tgz for binary.
> 
> https://kafka.apache.org/downloads.html
> 
> On Tue, Nov 29, 2016 at 5:40 PM, Gwen Shapira  wrote:
> 
>> Sorry for my misunderstanding, I assumed the request to include the
>> keyword removal came from you.
>> 
>> And it is always good to know what versions LinkedIn are running, you
>> guys always serve as somewhat of a gold standard to the community :)
>> 
>> On Tue, Nov 29, 2016 at 5:32 PM, Ignacio Solis  wrote:
>>> I don't think anybody from LinkedIn asked for features on this release.
>> We
>>> just jumped in at the discussion of including a patch which was not a bug
>>> fix and whether it mattered.
>>> 
>>> Having said that, the internal release we're working on came off the
>> 0.10.1
>>> branch with a few internal hotfix patches and a few cherry picked
>> fixes...
>>> Including the final keyword removal patch.
>>> 
>>> Nacho
>>> 
>>> 
>>> On Tue, Nov 29, 2016, 5:15 PM Gwen Shapira  wrote:
 
 btw. is LinkedIn no longer running from trunk? I'm not used to
 LinkedIn employees requesting specific patches to be included in a
 bugfix release.
 
 Any discussion on the content of any release is obviously welcome, I'm
 just wondering if there was a change in policy.
 
 On Tue, Nov 29, 2016 at 2:17 PM, Ismael Juma  wrote:
> OK, so it seems like there are no changes that break compatibility in
> the
> 0.10.1 branch since we offer no compatibility guarantees for logging
> output. That's good. :)
> 
> About the removal of final, it happened in trunk and it doesn't seem
> like
> anyone is still asking for it to be included in the 0.10.1 branch so
>> it
> is
> indeed not important for this bug fix release (I thought we had
> established
> that quite a while ago).
> 
> Ismael
> 
> On Tue, Nov 29, 2016 at 9:35 PM, Ignacio Solis 
>> wrote:
> 
>> Sorry, that was a hasty reply.  There are also various logging things
>> that
>> change format. This could break parsers.
>> 
>> None of them are important, my only argument is that the final
>> keyword
>> removal is not important either.
>> 
>> Nacho
>> 
>> 
>> On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis 
>> wrote:
>> 
>>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
>>> 10cfc1628df024f7596d3af5c168fa90f59035ca
>>> 
>>> On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma 
>>> wrote:
>>> 
 Which changes break compatibility in the 0.10.1 branch? It would
>> be
 good
 to
 fix before the release goes out.
 
 Ismael
 
 On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
 
> Some of the changes in the 0.10.1 branch already are not bug
> fixes.
>> Some
> break compatibility.
> 
> Having said that, at this level we should maintain a stable API
> and
 leave
> any changes for real version bumps.  This should be only a
>> bugfix
 release.
> 
> Nacho
> 
> 
> 
> 
> On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma >> 
>> wrote:
> 
>> I disagree, but let's discuss it another time and in a
>> separate
 thread.
> :)
>> 
>> Ismael
>> 
>> On Tue, Nov 29, 2016 at 4:30 PM, radai
>> 
> wrote:
>> 
>>> designing kafka code for stable extensibility is a worthy
>> and
>> noble
>> cause.
>>> however, seeing as there are no such derivatives out in the
>>> wild
 yet i
>>> think investing the effort right now is 

Re: [VOTE] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Ewen Cheslack-Postava]

+1 (binding).

Also, see my notes in discussion thread around future compatibility
discussions for breaking plugin interface changes like this.

-Ewen

On Tue, Nov 29, 2016 at 3:54 PM, Guozhang Wang  wrote:

> +1.
>
> On Tue, Nov 29, 2016 at 11:05 AM, Bill Bejeck  wrote:
>
> > +1 (non-binding)
> >
> > Thanks,
> >
> > Bill
> >
> > On Tue, Nov 29, 2016 at 1:34 PM, Matthias J. Sax 
> > wrote:
> >
> > > I’d like to start the voting process for KIP-93:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 93%3A+Improve+invalid+timestamp+handling+in+Kafka+Streams
> > >
> > > -Matthias
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>



-- 
Thanks,
Ewen

Re: [DISCUSS] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Ewen Cheslack-Postava]

I think this looks reasonable, but just a more general note on
compatibility -- I think it's worth trying to clarify what types of
compatibility we're trying to achieve. Guozhang's 1 & 2 give an important
breakdown (compile vs runtime compatibility). For the type of change
described here, I think it makes sense to clarify the compatibility goals.
The (pure) compile time compatibility vs (pure) runtime compatibility
aren't the only options -- you have some additional intermediate choices as
well.

The proposal describes a change which requires recompiling the plugin
(TimestampExtractor) *and* substituting a runtime library (kafka-streams
jar) to get correct updated behavior. This can get interesting if you
already have N streams apps sharing the same TimestampExtractor. You now
*must* update all of them to the new streams jar if any are to be updated
for the new TimestampExtractor API. For folks with a monolithic
repo/versioning setup, this could potentially be painful since they're
forced to update all apps at once. It's at least not too bad since it can
be isolated to a single commit (without deployment needing to be
coordinated, for example), but if the # of apps gets > 4 or 5, these types
of updates start to be a real pain.

I think this API change is an acceptable (albeit annoying) API
incompatibility right now, but wanted to raise this in the discussion of
this KIP so we consider this moving forward. There definitely are
alternatives that add the new functionality but maintain compatibility
better. In particular, it's possible to define the new interface to require
both APIs:

// new interface
public interface TimestampExtractor {
long extract(ConsumerRecord record);
long extract(ConsumerRecord record, long
previousTimestamp);
}

which requires more effort for the implementor of the new API, but
maintains compatibility if you want to use a new jar including the
TimestampExtractor even with the old version of streams/the
TimestampExtractor interface (since it will correctly dispatch to the old
implementation). It requires more effort on the part of the framework since
it needs to catch runtime exceptions when the second version of extract()
is missing and fall back to the first version. But in some cases that might
be warranted for the sake of compatibility.

I suspect this update won't cause too much pain right now just because the
number of streams app any user has won't be too large quite yet, but this
seems important to consider moving forward. I think we had some similar
concerns & discussion around the changes to the consumer APIs when trying
to generalize the collection types used in those APIs.

-Ewen


On Mon, Nov 28, 2016 at 10:46 AM, Matthias J. Sax 
wrote:

> Done.
>
> If there is no further comments, I would like to start a voting thread
> in a separate email.
>
> -Matthias
>
> On 11/28/16 9:08 AM, Guozhang Wang wrote:
> > Yes it does not include these, again in my previous previous email I
> meant
> > when you say "This is a breaking, incompatible change" people may
> interpret
> > it differently. So better explain it more clearly.
> >
> >
> > Guozhang
> >
> > On Thu, Nov 24, 2016 at 10:31 PM, Matthias J. Sax  >
> > wrote:
> >
> >> That does make sense. But KIP-93 does not change anything like this, so
> >> there is nothing to mention, IMHO.
> >>
> >> Or do you mean, the KIP should include that the change is backward
> >> compatible with this regard?
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 11/24/16 5:31 PM, Guozhang Wang wrote:
> >>> What I meant is that, for some changes (e.g. say we change the
> >>> auto-repartition behavior that caused using different name conventions,
> >> or
> >>> some changes that involve changing the underlying state store names,
> etc)
> >>> the existing internal state including the stores and topics will
> probably
> >>> not valid. Some users consider this also as a "backward incompatible
> >>> change" since they cannot just swipe in the new jar and restart.
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Wed, Nov 23, 2016 at 3:20 PM, Matthias J. Sax <
> matth...@confluent.io>
> >>> wrote:
> >>>
>  Thanks for the feedback. I updated the KIP for (1) and (2).
> 
>  However not for (3): Why should it be required to reset an
> application?
>  If user processed "good" data with valid timestamps, behavior does not
>  change. If user tried to process "bad" data with invalid timestamps,
> the
>  application does fail currently anyway, so there is nothing to reset.
> 
> 
>  -Matthias
> 
>  On 11/22/16 9:53 AM, Guozhang Wang wrote:
> > Regarding the "compatibility" section, I would suggest being a bit
> more
> > specific about why it is a breaking change. For Streams, it could
> mean
> > different things:
> >
> > 1. User need code change when switching library dependency on the new
> > version, otherwise it won't 

Re: [DISCUSS] 0.10.1.1 Plan

[Guozhang Wang]

@Ignacio Solis

The commit you mentioned was not intended for 0.10.1 but only for trunk
(and there is a related KIP for this API change), but mistakenly gets
leaked in and was already reverted.

@Bernard Leach

Could you elaborate on "instead simply publish the artifacts to maven
central"? Currently the Kafka release is already through maven and we do
not yet have kafka_2.12-0.10.1.0.tgz for binary.

https://kafka.apache.org/downloads.html

On Tue, Nov 29, 2016 at 5:40 PM, Gwen Shapira  wrote:

> Sorry for my misunderstanding, I assumed the request to include the
> keyword removal came from you.
>
> And it is always good to know what versions LinkedIn are running, you
> guys always serve as somewhat of a gold standard to the community :)
>
> On Tue, Nov 29, 2016 at 5:32 PM, Ignacio Solis  wrote:
> > I don't think anybody from LinkedIn asked for features on this release.
> We
> > just jumped in at the discussion of including a patch which was not a bug
> > fix and whether it mattered.
> >
> > Having said that, the internal release we're working on came off the
> 0.10.1
> > branch with a few internal hotfix patches and a few cherry picked
> fixes...
> > Including the final keyword removal patch.
> >
> > Nacho
> >
> >
> > On Tue, Nov 29, 2016, 5:15 PM Gwen Shapira  wrote:
> >>
> >> btw. is LinkedIn no longer running from trunk? I'm not used to
> >> LinkedIn employees requesting specific patches to be included in a
> >> bugfix release.
> >>
> >> Any discussion on the content of any release is obviously welcome, I'm
> >> just wondering if there was a change in policy.
> >>
> >> On Tue, Nov 29, 2016 at 2:17 PM, Ismael Juma  wrote:
> >> > OK, so it seems like there are no changes that break compatibility in
> >> > the
> >> > 0.10.1 branch since we offer no compatibility guarantees for logging
> >> > output. That's good. :)
> >> >
> >> > About the removal of final, it happened in trunk and it doesn't seem
> >> > like
> >> > anyone is still asking for it to be included in the 0.10.1 branch so
> it
> >> > is
> >> > indeed not important for this bug fix release (I thought we had
> >> > established
> >> > that quite a while ago).
> >> >
> >> > Ismael
> >> >
> >> > On Tue, Nov 29, 2016 at 9:35 PM, Ignacio Solis 
> wrote:
> >> >
> >> >> Sorry, that was a hasty reply.  There are also various logging things
> >> >> that
> >> >> change format. This could break parsers.
> >> >>
> >> >> None of them are important, my only argument is that the final
> keyword
> >> >> removal is not important either.
> >> >>
> >> >> Nacho
> >> >>
> >> >>
> >> >> On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis 
> wrote:
> >> >>
> >> >> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
> >> >> > 10cfc1628df024f7596d3af5c168fa90f59035ca
> >> >> >
> >> >> > On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma 
> >> >> > wrote:
> >> >> >
> >> >> >> Which changes break compatibility in the 0.10.1 branch? It would
> be
> >> >> >> good
> >> >> >> to
> >> >> >> fix before the release goes out.
> >> >> >>
> >> >> >> Ismael
> >> >> >>
> >> >> >> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
> >> >> >>
> >> >> >> > Some of the changes in the 0.10.1 branch already are not bug
> >> >> >> > fixes.
> >> >> Some
> >> >> >> > break compatibility.
> >> >> >> >
> >> >> >> > Having said that, at this level we should maintain a stable API
> >> >> >> > and
> >> >> >> leave
> >> >> >> > any changes for real version bumps.  This should be only a
> bugfix
> >> >> >> release.
> >> >> >> >
> >> >> >> > Nacho
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma  >
> >> >> wrote:
> >> >> >> >
> >> >> >> > > I disagree, but let's discuss it another time and in a
> separate
> >> >> >> thread.
> >> >> >> > :)
> >> >> >> > >
> >> >> >> > > Ismael
> >> >> >> > >
> >> >> >> > > On Tue, Nov 29, 2016 at 4:30 PM, radai
> >> >> >> > > 
> >> >> >> > wrote:
> >> >> >> > >
> >> >> >> > > > designing kafka code for stable extensibility is a worthy
> and
> >> >> noble
> >> >> >> > > cause.
> >> >> >> > > > however, seeing as there are no such derivatives out in the
> >> >> >> > > > wild
> >> >> >> yet i
> >> >> >> > > > think investing the effort right now is a bit premature from
> >> >> kafka's
> >> >> >> > pov.
> >> >> >> > > > I think its enough simply not to purposefully prevent such
> >> >> >> extensions.
> >> >> >> > > >
> >> >> >> > > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma
> >> >> >> > > > 
> >> >> >> > wrote:
> >> >> >> > > >
> >> >> >> > > > > On Sat, Nov 26, 2016 at 11:08 PM, radai <
> >> >> >> radai.rosenbl...@gmail.com>
> >> >> >> > > > > wrote:
> >> >> >> > > > >
> >> >> >> > > > > > "compatibility guarantees that are expected by people
> who
> >> >> >> subclass
> >> >> >> > > > these
> >> >> 

Jenkins build is back to normal : kafka-trunk-jdk8 #1065

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #2191: KAFKA-4447: Controller resigned but it also acts a...

[xiguantiaozhan]

GitHub user xiguantiaozhan opened a pull request:

https://github.com/apache/kafka/pull/2191

KAFKA-4447: Controller resigned but it also acts as a controller for a long 
time



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiguantiaozhan/kafka avoid_swamp_controllerLog

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2191


commit 3d5a9dce5f01d85aeaaf454e3471fda61d9c08a3
Author: tuyang 
Date:   2016-11-30T05:42:28Z

avoid swamping the old controller log




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707623#comment-15707623
 ] 

ASF GitHub Bot commented on KAFKA-4447:
---

GitHub user xiguantiaozhan opened a pull request:

https://github.com/apache/kafka/pull/2191

KAFKA-4447: Controller resigned but it also acts as a controller for a long 
time



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiguantiaozhan/kafka avoid_swamp_controllerLog

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2191


commit 3d5a9dce5f01d85aeaaf454e3471fda61d9c08a3
Author: tuyang 
Date:   2016-11-30T05:42:28Z

avoid swamping the old controller log




> Controller resigned but it also acts as a controller for a long time 
> -
>
> Key: KAFKA-4447
> URL: https://issues.apache.org/jira/browse/KAFKA-4447
> Project: Kafka
>  Issue Type: Improvement
>  Components: controller
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
> Environment: Linux Os
>Reporter: Json Tu
> Attachments: log.tar.gz
>
>
> We have a cluster with 10 nodes，and we execute following operation as below.
> 1.we execute some topic partition reassign from one node to other 9 nodes in 
> the cluster, and which triggered controller.
> 2.controller invoke PartitionsReassignedListener's handleDataChange and read 
> all partition reassign rules from the zk path, and executed all 
> onPartitionReassignment for all partition that match conditions.
> 3.but the controller is expired from zk, after what some nodes of 9 nodes 
> also expired from zk.
> 5.then controller invoke onControllerResignation to resigned as the 
> controller.
> we found after the controller is resigned, it acts as controller for about 3 
> minutes, which can be found in my attachment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

[Matthias J. Sax]

Thanks for your input.

To clarify: the main reason to add the metadata topic is to cope with
subtopologies that are connected via intermediate topic (either
user-defined via through() or internally created for data repartitioning).

Without this handling, the behavior would be odd and user experience
would be bad.

Thus, using the metadata topic for have a "fixed HW" is just a small
add-on -- and more or less for free, because the metadata topic is
already there.


-Matthias


On 11/29/16 7:53 PM, Neha Narkhede wrote:
> Thanks for initiating this. I think this is a good first step towards
> unifying batch and stream processing in Kafka.
> 
> I understood this capability to be simple yet very useful; it allows a
> Streams program to process a log, in batch, in arbitrary windows defined by
> the difference between the HW and the current offset. Basically, it
> provides a simple means for a Streams program to "stop" after processing a
> batch, stop (just like a batch program would) and continue where it left
> off when restarted. In other words, it allows batch processing behavior for
> a Streams app without code changes.
> 
> This feature is useful but I do not think there is a necessity to add a
> metadata topic. After all, the user doesn't really care as much about
> exactly where the batch ends. This feature allows an app to "process as
> much as there is data to process" and the way it determines how much data
> there is to process is by reading the HW on startup. If there is new data
> written to the log right after it starts up, it will process it when
> restarted the next time. If it starts, reads HW but fails, it will restart
> and process a little more before it stops again. The fact that the HW
> changes in some scenarios isn't an issue since a batch program that behaves
> this way doesn't really care exactly what that HW is.
> 
> There might be cases which require adding more topics but I would shy away
> from adding complexity wherever possible as it complicates operations and
> reduces simplicity.
> 
> Other than this issue, I'm +1 on adding this feature. I think it is pretty
> powerful.
> 
> 
> On Mon, Nov 28, 2016 at 10:48 AM Matthias J. Sax 
> wrote:
> 
>> Hi all,
>>
>> I want to start a discussion about KIP-95:
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams
>>
>> Looking forward to your feedback.
>>
>>
>> -Matthias
>>
>>
>> --
> Thanks,
> Neha
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Resolved] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin resolved KAFKA-3042.
-
Resolution: Duplicate
  Assignee: Dong Lin

The problem may have been solved in KAFKA-4443.

> updateIsr should stop after failed several times due to zkVersion issue
> ---
>
> Key: KAFKA-3042
> URL: https://issues.apache.org/jira/browse/KAFKA-3042
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
> Environment: jdk 1.7
> centos 6.4
>Reporter: Jiahongchao
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.2.0
>
> Attachments: controller.log, server.log.2016-03-23-01, 
> state-change.log
>
>
> sometimes one broker may repeatly log
> "Cached zkVersion 54 not equal to that in zookeeper, skip updating ISR"
> I think this is because the broker consider itself as the leader in fact it's 
> a follower.
> So after several failed tries, it need to find out who is the leader



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4463) Setup travis-ci integration for ducktape tests

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4463:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Setup travis-ci integration for ducktape tests
> --
>
> Key: KAFKA-4463
> URL: https://issues.apache.org/jira/browse/KAFKA-4463
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.10.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4466) Add support to ducktape to run only a part of all tests

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4466:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Add support to ducktape to run only a part of all tests
> ---
>
> Key: KAFKA-4466
> URL: https://issues.apache.org/jira/browse/KAFKA-4466
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4467) Run tests on travis-ci using docker

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4467:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Run tests on travis-ci using docker
> ---
>
> Key: KAFKA-4467
> URL: https://issues.apache.org/jira/browse/KAFKA-4467
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4465) Create docker image and scripts for running tests locally

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4465:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Create docker image and scripts for running tests locally
> -
>
> Key: KAFKA-4465
> URL: https://issues.apache.org/jira/browse/KAFKA-4465
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4387:
---
   Resolution: Fixed
Fix Version/s: 0.10.2.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2110
[https://github.com/apache/kafka/pull/2110]

> KafkaConsumer will enter an infinite loop if the polling thread is 
> interrupted, and either commitSync or committed is called
> 
>
> Key: KAFKA-4387
> URL: https://issues.apache.org/jira/browse/KAFKA-4387
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
> Fix For: 0.10.2.0
>
>
> When the KafkaConsumer.commitSync method is called, the 
> ConsumerNetworkClient.poll(RequestFuture future) method will be called 
> with a future that only finishes when the commit request completes, or the 
> request times out.
> When the calling thread is interrupted, every call to the Selector underlying 
> the ConsumerNetworkClient will return immediately, while thread interrupt 
> state is not reset. The call to poll ends up looping until the request 
> timeout, at which point it drops back out to 
> ConsumerCoordinator.commitOffsetsSync which retries the request because 
> TimeoutException is retriable. This repeats indefinitely. 
> For the same reason as in https://issues.apache.org/jira/browse/KAFKA-4375, 
> it is good if the KafkaConsumer can handle interrupts in a reasonable way, 
> rather than having wakeup() be the only way to properly stop a consumer 
> thread.
> I think making ConsumerNetworkClient.maybeTriggerWakeup() throw a 
> WakeupException if the calling thread is interrupted makes sense, since an 
> interrupted thread won't be making progress in polling due to the way 
> Selector works, and KafkaConsumer users then don't have to handle wakeups and 
> interrupts separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707489#comment-15707489
 ] 

ASF GitHub Bot commented on KAFKA-4387:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2110


> KafkaConsumer will enter an infinite loop if the polling thread is 
> interrupted, and either commitSync or committed is called
> 
>
> Key: KAFKA-4387
> URL: https://issues.apache.org/jira/browse/KAFKA-4387
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
> Fix For: 0.10.2.0
>
>
> When the KafkaConsumer.commitSync method is called, the 
> ConsumerNetworkClient.poll(RequestFuture future) method will be called 
> with a future that only finishes when the commit request completes, or the 
> request times out.
> When the calling thread is interrupted, every call to the Selector underlying 
> the ConsumerNetworkClient will return immediately, while thread interrupt 
> state is not reset. The call to poll ends up looping until the request 
> timeout, at which point it drops back out to 
> ConsumerCoordinator.commitOffsetsSync which retries the request because 
> TimeoutException is retriable. This repeats indefinitely. 
> For the same reason as in https://issues.apache.org/jira/browse/KAFKA-4375, 
> it is good if the KafkaConsumer can handle interrupts in a reasonable way, 
> rather than having wakeup() be the only way to properly stop a consumer 
> thread.
> I think making ConsumerNetworkClient.maybeTriggerWakeup() throw a 
> WakeupException if the calling thread is interrupted makes sense, since an 
> interrupted thread won't be making progress in polling due to the way 
> Selector works, and KafkaConsumer users then don't have to handle wakeups and 
> interrupts separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2110: KAFKA-4387: Fix KafkaConsumer not responding corre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2110


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-96 - Add per partition metrics for in-sync and assigned replica count

[Neha Narkhede]

This seems useful, +1

On Tue, Nov 29, 2016 at 5:39 AM Ismael Juma  wrote:

> Hi Xavier,
>
> Thanks for the KIP. Sounds good to me.
>
> Ismael
>
> On Tue, Nov 29, 2016 at 12:40 AM, Xavier Léauté 
> wrote:
>
> > Hi,
> >
> > I created KIP-96 to propose per partition in-sync / assigned replica
> > metrics. Should be straightforward, but submitting it for proposal since
> we
> > require it for metrics changes.
> >
> > Here's the link to the KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count
> >
> > Thank you,
> > Xavier
> >
>
-- 
Thanks,
Neha

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

[Neha Narkhede]

Thanks for initiating this. I think this is a good first step towards
unifying batch and stream processing in Kafka.

I understood this capability to be simple yet very useful; it allows a
Streams program to process a log, in batch, in arbitrary windows defined by
the difference between the HW and the current offset. Basically, it
provides a simple means for a Streams program to "stop" after processing a
batch, stop (just like a batch program would) and continue where it left
off when restarted. In other words, it allows batch processing behavior for
a Streams app without code changes.

This feature is useful but I do not think there is a necessity to add a
metadata topic. After all, the user doesn't really care as much about
exactly where the batch ends. This feature allows an app to "process as
much as there is data to process" and the way it determines how much data
there is to process is by reading the HW on startup. If there is new data
written to the log right after it starts up, it will process it when
restarted the next time. If it starts, reads HW but fails, it will restart
and process a little more before it stops again. The fact that the HW
changes in some scenarios isn't an issue since a batch program that behaves
this way doesn't really care exactly what that HW is.

There might be cases which require adding more topics but I would shy away
from adding complexity wherever possible as it complicates operations and
reduces simplicity.

Other than this issue, I'm +1 on adding this feature. I think it is pretty
powerful.


On Mon, Nov 28, 2016 at 10:48 AM Matthias J. Sax 
wrote:

> Hi all,
>
> I want to start a discussion about KIP-95:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams
>
> Looking forward to your feedback.
>
>
> -Matthias
>
>
> --
Thanks,
Neha

Re: [VOTE] KIP-85: Dynamic JAAS configuration for Kafka clients

[Neha Narkhede]

Super useful, thanks Rajini. +1

On Tue, Nov 29, 2016 at 8:11 AM Rajini Sivaram 
wrote:

> I have added this to the KIP.
>
> Thanks,
>
> Rajini
>
> On Tue, Nov 29, 2016 at 12:14 PM, Ismael Juma  wrote:
>
> > One more thing: we are using the PASSWORD config type to avoid exposing
> > passwords. This will also make it harder to debug issues with the JAAS
> > config though, it would be good to mention this drawback in the KIP.
> >
> > Ismael
> >
> > On Mon, Nov 28, 2016 at 1:00 PM, Ismael Juma  wrote:
> >
> > > I'm very late to this, but better late than never, I guess. I am +1 on
> > > this because it improves on the status quo, satisfies a real need and
> is
> > > simple to implement.
> > >
> > > Having said that, I'd also like to state that it's a bit of a shame
> that
> > > we are doubling down on the JAAS config format. It is a peculiar format
> > and
> > > in the Kerberos case (one of the common usages), it requires users to
> > > provide different configs depending on the Java implementation being
> > used.
> > > It would be nice if we looked into abstracting some of this to make
> > users'
> > > lives easier. Looking at the Hadoop codebase, it looks like they try to
> > do
> > > that although I don't know how well it worked out in practice.
> > >
> > > Ismael
> > >
> > > On Tue, Nov 1, 2016 at 1:42 PM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > >> KIP-85 vote has passed with 4 binding (Harsha, Gwen, Jason, Jun) and 4
> > >> non-binding (Mickael, Jim, Edo, me) votes.
> > >>
> > >> Thank you all for your votes and comments. I will update the KIP page
> > and
> > >> rebase the PR.
> > >>
> > >> Many thanks,
> > >>
> > >> Rajini
> > >>
> > >>
> > >>
> > >> On Mon, Oct 31, 2016 at 11:29 AM, Edoardo Comar 
> > >> wrote:
> > >>
> > >> > +1 great KIP
> > >> > --
> > >> > Edoardo Comar
> > >> > IBM MessageHub
> > >> > eco...@uk.ibm.com
> > >> > IBM UK Ltd, Hursley Park, SO21 2JN
> > >> >
> > >> > IBM United Kingdom Limited Registered in England and Wales with
> number
> > >> > 741598 Registered office: PO Box 41, North Harbour, Portsmouth,
> Hants.
> > >> PO6
> > >> > 3AU
> > >> >
> > >> >
> > >> >
> > >> > From:   Rajini Sivaram 
> > >> > To: dev@kafka.apache.org
> > >> > Date:   26/10/2016 16:27
> > >> > Subject:[VOTE] KIP-85: Dynamic JAAS configuration for Kafka
> > >> > clients
> > >> >
> > >> >
> > >> >
> > >> > I would like to initiate the voting process for KIP-85: Dynamic JAAS
> > >> > configuration for Kafka Clients:
> > >> >
> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+
> > >> Dynamic+JAAS+
> > >> > configuration+for+Kafka+clients
> > >> >
> > >> >
> > >> > This KIP enables Java clients to connect to Kafka using SASL
> without a
> > >> > physical jaas.conf file. This will also be useful to configure
> > multiple
> > >> > KafkaClient login contexts when multiple users are supported within
> a
> > >> JVM.
> > >> >
> > >> > Thank you...
> > >> >
> > >> > Regards,
> > >> >
> > >> > Rajini
> > >> >
> > >> >
> > >> >
> > >> > Unless stated otherwise above:
> > >> > IBM United Kingdom Limited - Registered in England and Wales with
> > number
> > >> > 741598.
> > >> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
> PO6
> > >> 3AU
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Regards,
> > >>
> > >> Rajini
> > >>
> > >
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
-- 
Thanks,
Neha

[jira] [Commented] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707407#comment-15707407
 ] 

Dong Lin commented on KAFKA-3042:
-

We recently made a fix in Kafka that may have the addressed the problem 
mentioned this JIRA. See KAFKA-4443 for more information.

> updateIsr should stop after failed several times due to zkVersion issue
> ---
>
> Key: KAFKA-3042
> URL: https://issues.apache.org/jira/browse/KAFKA-3042
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
> Environment: jdk 1.7
> centos 6.4
>Reporter: Jiahongchao
>  Labels: reliability
> Fix For: 0.10.2.0
>
> Attachments: controller.log, server.log.2016-03-23-01, 
> state-change.log
>
>
> sometimes one broker may repeatly log
> "Cached zkVersion 54 not equal to that in zookeeper, skip updating ISR"
> I think this is because the broker consider itself as the leader in fact it's 
> a follower.
> So after several failed tries, it need to find out who is the leader



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4403) Update KafkaBasedLog to use new endOffsets consumer API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707381#comment-15707381
 ] 

ASF GitHub Bot commented on KAFKA-4403:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2176


> Update KafkaBasedLog to use new endOffsets consumer API
> ---
>
> Key: KAFKA-4403
> URL: https://issues.apache.org/jira/browse/KAFKA-4403
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Balint Molnar
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.2.0
>
>
> As of 0.10.1.0 and KIP-79 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868090) 
> KafkaConsumer can now fetch offset information about topic partitions. 
> Previously KafkaBasedLog had to use a seekToEnd + position approach to 
> determine end offsets. With the new APIs we can simplify this code.
> This isn't critical as the current code works fine, but would be a nice 
> cleanup and simplification.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4403) Update KafkaBasedLog to use new endOffsets consumer API

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-4403.

   Resolution: Fixed
Fix Version/s: 0.10.2.0

Issue resolved by pull request 2176
[https://github.com/apache/kafka/pull/2176]

> Update KafkaBasedLog to use new endOffsets consumer API
> ---
>
> Key: KAFKA-4403
> URL: https://issues.apache.org/jira/browse/KAFKA-4403
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Balint Molnar
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.2.0
>
>
> As of 0.10.1.0 and KIP-79 
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868090) 
> KafkaConsumer can now fetch offset information about topic partitions. 
> Previously KafkaBasedLog had to use a seekToEnd + position approach to 
> determine end offsets. With the new APIs we can simplify this code.
> This isn't critical as the current code works fine, but would be a nice 
> cleanup and simplification.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2176: KAFKA-4403 Update KafkaBasedLog to use new endOffs...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2176


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3893) Kafka Broker ID disappears from /brokers/ids

[Rekha Joshi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rekha Joshi updated KAFKA-3893:
---
Summary: Kafka Broker ID disappears from /brokers/ids  (was: Kafka Broker 
ID disappears from /borkers/ids)

> Kafka Broker ID disappears from /brokers/ids
> 
>
> Key: KAFKA-3893
> URL: https://issues.apache.org/jira/browse/KAFKA-3893
> Project: Kafka
>  Issue Type: Bug
>Reporter: chaitra
>Priority: Critical
>
> Kafka version used : 0.8.2.1 
> Zookeeper version: 3.4.6
> We have scenario where kafka 's broker in  zookeeper path /brokers/ids just 
> disappears.
> We see the zookeeper connection active and no network issue.
> The zookeeper conection timeout is set to 6000ms in server.properties
> Hence Kafka not participating in cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3893) Kafka Broker ID disappears from /borkers/ids

[Rekha Joshi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rekha Joshi updated KAFKA-3893:
---
Summary: Kafka Broker ID disappears from /borkers/ids  (was: Kafka Borker 
ID disappears from /borkers/ids)

> Kafka Broker ID disappears from /borkers/ids
> 
>
> Key: KAFKA-3893
> URL: https://issues.apache.org/jira/browse/KAFKA-3893
> Project: Kafka
>  Issue Type: Bug
>Reporter: chaitra
>Priority: Critical
>
> Kafka version used : 0.8.2.1 
> Zookeeper version: 3.4.6
> We have scenario where kafka 's broker in  zookeeper path /brokers/ids just 
> disappears.
> We see the zookeeper connection active and no network issue.
> The zookeeper conection timeout is set to 6000ms in server.properties
> Hence Kafka not participating in cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3893) Kafka Borker ID disappears from /borkers/ids

[Rekha Joshi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15616914#comment-15616914
 ] 

Rekha Joshi edited comment on KAFKA-3893 at 11/30/16 2:58 AM:
--

We saw it too in Kafka 0.8.x and upgraded to Kafka 0.9.x and continued seeing 
this issue, especially in AWS.This was seen in AWS especially if you configure 
zookeeper connect property with zookeeper ELB instead of individual IP:port 
set. When we reversed zookeeper connect with zookeeper IP:port instead of 
zookeeper ELB, it worked great.
It would be great(can help) if Apache Kafka/Confluent FAQ can be updated to 
note this.
Thanks
Rekha



was (Author: rekhajoshm):
We saw it too in Kafka 0.8.x and upgraded to Kafka 0.9.x and continued seeing 
this issue.Going with workarounds.But it would be great(can help) if this is 
documented clearly on Apache Kafka/Confluent FAQ.
Thanks
Rekha


> Kafka Borker ID disappears from /borkers/ids
> 
>
> Key: KAFKA-3893
> URL: https://issues.apache.org/jira/browse/KAFKA-3893
> Project: Kafka
>  Issue Type: Bug
>Reporter: chaitra
>Priority: Critical
>
> Kafka version used : 0.8.2.1 
> Zookeeper version: 3.4.6
> We have scenario where kafka 's broker in  zookeeper path /brokers/ids just 
> disappears.
> We see the zookeeper connection active and no network issue.
> The zookeeper conection timeout is set to 6000ms in server.properties
> Hence Kafka not participating in cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707330#comment-15707330
 ] 

Dong Lin edited comment on KAFKA-4443 at 11/30/16 2:55 AM:
---

[~junrao] Sure. I just updated the description to correct the typo. What I mean 
is that, if a broker starts right after controller election, the 
LeaderAndIsrRequest will be ignored because the broker doesn't have the needed 
information (e.g. port) of live brokers.

As for (2), I think this is probably the same issue reported in KAFKA-3042. All 
phenomena described in KAFKA-3042 can be caused by the bug fixed in this JIRA. 
Actually, you described exactly the same fix applied in this JIRA 7 months ago, 
i.e. "... to fix this particular issue, the simplest approach is to send 
UpdateMetadataRequest first during controller failover".

As of current design of controller, I prefer the solution where controller 
sends MetadataUpdateRequest without LeaderAndIsrRequset. Broker will handle 
MedataDataUpdateRequest in the following steps: 1) update cache with live 
broker info extracted from MetadataUpdateRequest, 2) reconstruct 
LeaderAndIsrRequest from MetadataUpdateRequest and process it, and 3) update 
cache with partition information extracted from MetadataUpdateRequest. This 
solution is simple and doesn't require wire protocol change. And it is strictly 
better than current implementation because we no longer have to send 
MetadataUpdateRequest before LeaderAndIsrRequest. 

But I am not 100% sure this is long term solution because it relies on existing 
implementation detail where controller always send MetadataUpdateRequest after 
LeaderAndIsrRequest. In theory this may not be the case if controller is 
re-designed. For example, we may want to send MetadataUpdateRequest only after 
Controller has received LeaderAndIsrResponse with success. The idea is to 
expose new external state to user only after internal state change is completed.

If we don't adopt the solution above which uses MetadataUpateRequest as 
combination of LeaderAndIsrRequest + MetadataUpdateRequest, then I think we 
should include endpoints of all leaders in the LeaderAndIsrRequest so that 
LeaderAndIsrRequest can provide enough information on its own to switch broker 
between leader and follower.


was (Author: lindong):
[~junrao] Sure. I just updated the description to correct the typo. What I mean 
is that, if a broker starts right after controller election, the 
LeaderAndIsrRequest will be ignored because the broker doesn't have the needed 
information (e.g. port) of live brokers.

As for (2), I think this is probably the same issue reported in KAFKA-3042. All 
phenomena described in KAFKA-3042 can be caused by the bug fixed in this JIRA. 
Actually, you described exactly the same fix applied in this JIRA 7 months ago, 
i.e. "... to fix this particular issue, the simplest approach is to send 
UpdateMetadataRequest first during controller failover".

As of current design of controller, I prefer the solution where controller 
sends MetadataUpdateRequest without LeaderAndIsrRequset. Broker will handle 
MedataDataUpdateRequest in the following steps: 1) update cache with live 
broker info extracted from MetadataUpdateRequest, 2) reconstruct 
LeaderAndIsrRequest from MetadataUpdateRequest and process it, and 3) update 
cache with partition information extracted from MetadataUpdateRequest. This 
solution is simple and doesn't require wire protocol change. And it is strictly 
better than current implementation because we no longer have to send 
MetadataUpdateRequest before LeaderAndIsrRequest. 

But I am not 100% sure this is long term solution because it relies on existing 
implementation detail where controller always send MetadataUpdateRequest after 
LeaderAndIsrRequest. In theory this may not be the case if controller is 
re-designed. For example, we may want to send MetadataUpdateRequest only after 
Controller has received LeaderAndIsrResponse with success. The idea is to 
expose new external state to user only after internal state change is completed.

> Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest 
> during failover
> -
>
> Key: KAFKA-4443
> URL: https://issues.apache.org/jira/browse/KAFKA-4443
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.1.1
>
>
> Currently in onControllerFailover(), controller will startup 
> replicaStatemachine and partitionStateMachine before invoking 
> sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq).
>  However, if a broker starts right after controller election, the 
> LeaderAndIsrRequest sent 

[jira] [Commented] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707330#comment-15707330
 ] 

Dong Lin commented on KAFKA-4443:
-

[~junrao] Sure. I just updated the description to correct the typo. What I mean 
is that, if a broker starts right after controller election, the 
LeaderAndIsrRequest will be ignored because the broker doesn't have the needed 
information (e.g. port) of live brokers.

As for (2), I think this is probably the same issue reported in KAFKA-3042. All 
phenomena described in KAFKA-3042 can be caused by the bug fixed in this JIRA. 
Actually, you described exactly the same fix applied in this JIRA 7 months ago, 
i.e. "... to fix this particular issue, the simplest approach is to send 
UpdateMetadataRequest first during controller failover".

As of current design of controller, I prefer the solution where controller 
sends MetadataUpdateRequest without LeaderAndIsrRequset. Broker will handle 
MedataDataUpdateRequest in the following steps: 1) update cache with live 
broker info extracted from MetadataUpdateRequest, 2) reconstruct 
LeaderAndIsrRequest from MetadataUpdateRequest and process it, and 3) update 
cache with partition information extracted from MetadataUpdateRequest. This 
solution is simple and doesn't require wire protocol change. And it is strictly 
better than current implementation because we no longer have to send 
MetadataUpdateRequest before LeaderAndIsrRequest. 

But I am not 100% sure this is long term solution because it relies on existing 
implementation detail where controller always send MetadataUpdateRequest after 
LeaderAndIsrRequest. In theory this may not be the case if controller is 
re-designed. For example, we may want to send MetadataUpdateRequest only after 
Controller has received LeaderAndIsrResponse with success. The idea is to 
expose new external state to user only after internal state change is completed.

> Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest 
> during failover
> -
>
> Key: KAFKA-4443
> URL: https://issues.apache.org/jira/browse/KAFKA-4443
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.1.1
>
>
> Currently in onControllerFailover(), controller will startup 
> replicaStatemachine and partitionStateMachine before invoking 
> sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq).
>  However, if a broker starts right after controller election, the 
> LeaderAndIsrRequest sent to follower partitions on this broker will all be 
> ignored because broker doesn't know the leaders are alive. 
> To fix this problem, in onControllerFailover(), controller should send 
> UpdateMetadataRequest to brokers after initializeControllerContext() but 
> before it starts replicaStatemachine and partitionStateMachine. The first 
> MetadatUpdateRequest will include list of live broker. Although it will not 
> include partition leader information, it is OK because we will always send 
> MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
> replicaStateMachine.startup() and partitionStateMachine.startup().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4469) Consumer throughput regression caused by decrease in max.poll.records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707289#comment-15707289
 ] 

ASF GitHub Bot commented on KAFKA-4469:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2190

KAFKA-4469: Fix consumer performance regression from unneeded list copy



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4469

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2190


commit 701073de6163e66bac45ebb12130f3789f507531
Author: Jason Gustafson 
Date:   2016-11-30T01:50:32Z

KAFKA-4469: Fix consumer performance regression from unneeded list copy




> Consumer throughput regression caused by decrease in max.poll.records
> -
>
> Key: KAFKA-4469
> URL: https://issues.apache.org/jira/browse/KAFKA-4469
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.1.1
>
>
> There appears to be a small performance regression in 0.10.1.0 from previous 
> versions. I tracked it back to KAFKA-3888. As part of KIP-62, we decreased 
> the value of {{max.poll.records}} from {{Integer.MAX_VALUE}} to 500. Based on 
> some performance testing, this results in about a 5% decrease in throughput. 
> This depends on the fetch and message sizes. My test used message size of 1K 
> with the default fetch size, and the default {{max.poll.records}} of 500. 
> The main cause of the regression seems to be an unneeded list copy in 
> {{Fetcher}}. Basically when we have more records than we need to satisfy 
> {{max.poll.records}}, then we copy the fetched records into a new list. When 
> I modified the code to use a sub-list, which does not need a copy, the 
> performance is much closer to that of 0.10.0 (within 1% or so with lots of 
> qualification since there are many unexplored parameters).  The remaining 
> performance gap could be explained by sub-optimal pipelining as a result of 
> KAFKA-4007 (this is likely part of the story anyway based on some rough 
> testing).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2190: KAFKA-4469: Fix consumer performance regression fr...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2190

KAFKA-4469: Fix consumer performance regression from unneeded list copy



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4469

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2190


commit 701073de6163e66bac45ebb12130f3789f507531
Author: Jason Gustafson 
Date:   2016-11-30T01:50:32Z

KAFKA-4469: Fix consumer performance regression from unneeded list copy




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Brady Vidovic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707226#comment-15707226
 ] 

Brady Vidovic commented on KAFKA-1194:
--

I have downloaded the build and will test it out. Thanks for all your hard work 
on this!

> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-1194.patch, Untitled.jpg, kafka-1194-v1.patch, 
> kafka-1194-v2.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 1516723
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>  at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
>  at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
>  at scala.collection.immutable.List.foreach(List.scala:76)
>  at kafka.log.Log.deleteOldSegments(Log.scala:418)
>  at 
> kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>  at 
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
>  at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>  at 
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
>  at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>  at kafka.log.LogManager.cleanupLogs(LogManager.scala:314)
>  at 
> kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143)
>  at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
>  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:724)
> I think this error happens because kafka tries to rename the log file when it 
> is still opened.  So we should close the file first before rename.
> The index file uses a special data structure, the MappedByteBuffer. Javadoc 
> describes it as:
> A mapped byte buffer and the file mapping that it represents remain valid 
> until the buffer itself is garbage-collected.
> Fortunately, I find a forceUnmap function in kafka code, and perhaps it can 
> be used to free the MappedByteBuffer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4405) Kafka consumer improperly send prefetch request

[ysysberserk (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707215#comment-15707215
 ] 

ysysberserk edited comment on KAFKA-4405 at 11/30/16 1:52 AM:
--

Oh, I am sorry that I only read the code and did not notice that in 
fetchablePartitions() the partition which has records is removed before sent a 
prefetch.

So there is no such a big problem and current work flow is ok.

But we can still add check if total number of retained records is less than 
max.poll.records  before we call 

fetcher.sendFetches().It will prevent useless call and save a lot of time and 
make code eaiser to understand.


was (Author: ysysberserk):
Oh, I am sorry that I only read the code and did not notice that in 
fetchablePartitions() the partition which has records is removed before sent a 
prefetch.

So there is no such a big problem and current work flow is ok.

But we can still add check if total number of retained records is less than 
max.poll.records  before we call 

fetcher.sendFetches().It will prevent useless call and save a lot of time.

> Kafka consumer improperly send prefetch request
> ---
>
> Key: KAFKA-4405
> URL: https://issues.apache.org/jira/browse/KAFKA-4405
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: ysysberserk
>
> Now kafka consumer has added max.poll.records to limit the count of messages 
> return by poll().
> According to KIP-41, to implement  max.poll.records, the prefetch request 
> should only be sent when the total number of retained records is less than 
> max.poll.records.
> But in the code of 0.10.0.1 , the consumer will send a prefetch request if it 
> retained any records and never check if total number of retained records is 
> less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message 
> fetched , the poll() loop will send a lot of requests than expected and will 
> have more and more records fetched and stored in memory before they can be 
> consumed.
> So before sending a  prefetch request , the consumer must check if total 
> number of retained records is less than max.poll.records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

[ysysberserk (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707215#comment-15707215
 ] 

ysysberserk commented on KAFKA-4405:


Oh, I am sorry that I only read the code and did not notice that in 
fetchablePartitions() the partition which has records is removed before sent a 
prefetch.

So there is no such a big problem and current work flow is ok.

But we can still add check if total number of retained records is less than 
max.poll.records  before we call 

fetcher.sendFetches().It will prevent useless call and save a lot of time.

> Kafka consumer improperly send prefetch request
> ---
>
> Key: KAFKA-4405
> URL: https://issues.apache.org/jira/browse/KAFKA-4405
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: ysysberserk
>
> Now kafka consumer has added max.poll.records to limit the count of messages 
> return by poll().
> According to KIP-41, to implement  max.poll.records, the prefetch request 
> should only be sent when the total number of retained records is less than 
> max.poll.records.
> But in the code of 0.10.0.1 , the consumer will send a prefetch request if it 
> retained any records and never check if total number of retained records is 
> less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message 
> fetched , the poll() loop will send a lot of requests than expected and will 
> have more and more records fetched and stored in memory before they can be 
> consumed.
> So before sending a  prefetch request , the consumer must check if total 
> number of retained records is less than max.poll.records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4469) Consumer throughput regression caused by decrease in max.poll.records

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4469:
--

 Summary: Consumer throughput regression caused by decrease in 
max.poll.records
 Key: KAFKA-4469
 URL: https://issues.apache.org/jira/browse/KAFKA-4469
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.1.0
Reporter: Jason Gustafson
Assignee: Jason Gustafson
 Fix For: 0.10.1.1


There appears to be a small performance regression in 0.10.1.0 from previous 
versions. I tracked it back to KAFKA-3888. As part of KIP-62, we decreased the 
value of {{max.poll.records}} from {{Integer.MAX_VALUE}} to 500. Based on some 
performance testing, this results in about a 5% decrease in throughput. This 
depends on the fetch and message sizes. My test used message size of 1K with 
the default fetch size, and the default {{max.poll.records}} of 500. 

The main cause of the regression seems to be an unneeded list copy in 
{{Fetcher}}. Basically when we have more records than we need to satisfy 
{{max.poll.records}}, then we copy the fetched records into a new list. When I 
modified the code to use a sub-list, which does not need a copy, the 
performance is much closer to that of 0.10.0 (within 1% or so with lots of 
qualification since there are many unexplored parameters).  The remaining 
performance gap could be explained by sub-optimal pipelining as a result of 
KAFKA-4007 (this is likely part of the story anyway based on some rough 
testing).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 0.10.1.1 Plan

[Gwen Shapira]

Sorry for my misunderstanding, I assumed the request to include the
keyword removal came from you.

And it is always good to know what versions LinkedIn are running, you
guys always serve as somewhat of a gold standard to the community :)

On Tue, Nov 29, 2016 at 5:32 PM, Ignacio Solis  wrote:
> I don't think anybody from LinkedIn asked for features on this release.  We
> just jumped in at the discussion of including a patch which was not a bug
> fix and whether it mattered.
>
> Having said that, the internal release we're working on came off the 0.10.1
> branch with a few internal hotfix patches and a few cherry picked fixes...
> Including the final keyword removal patch.
>
> Nacho
>
>
> On Tue, Nov 29, 2016, 5:15 PM Gwen Shapira  wrote:
>>
>> btw. is LinkedIn no longer running from trunk? I'm not used to
>> LinkedIn employees requesting specific patches to be included in a
>> bugfix release.
>>
>> Any discussion on the content of any release is obviously welcome, I'm
>> just wondering if there was a change in policy.
>>
>> On Tue, Nov 29, 2016 at 2:17 PM, Ismael Juma  wrote:
>> > OK, so it seems like there are no changes that break compatibility in
>> > the
>> > 0.10.1 branch since we offer no compatibility guarantees for logging
>> > output. That's good. :)
>> >
>> > About the removal of final, it happened in trunk and it doesn't seem
>> > like
>> > anyone is still asking for it to be included in the 0.10.1 branch so it
>> > is
>> > indeed not important for this bug fix release (I thought we had
>> > established
>> > that quite a while ago).
>> >
>> > Ismael
>> >
>> > On Tue, Nov 29, 2016 at 9:35 PM, Ignacio Solis  wrote:
>> >
>> >> Sorry, that was a hasty reply.  There are also various logging things
>> >> that
>> >> change format. This could break parsers.
>> >>
>> >> None of them are important, my only argument is that the final keyword
>> >> removal is not important either.
>> >>
>> >> Nacho
>> >>
>> >>
>> >> On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis  wrote:
>> >>
>> >> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
>> >> > 10cfc1628df024f7596d3af5c168fa90f59035ca
>> >> >
>> >> > On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma 
>> >> > wrote:
>> >> >
>> >> >> Which changes break compatibility in the 0.10.1 branch? It would be
>> >> >> good
>> >> >> to
>> >> >> fix before the release goes out.
>> >> >>
>> >> >> Ismael
>> >> >>
>> >> >> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
>> >> >>
>> >> >> > Some of the changes in the 0.10.1 branch already are not bug
>> >> >> > fixes.
>> >> Some
>> >> >> > break compatibility.
>> >> >> >
>> >> >> > Having said that, at this level we should maintain a stable API
>> >> >> > and
>> >> >> leave
>> >> >> > any changes for real version bumps.  This should be only a bugfix
>> >> >> release.
>> >> >> >
>> >> >> > Nacho
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma 
>> >> wrote:
>> >> >> >
>> >> >> > > I disagree, but let's discuss it another time and in a separate
>> >> >> thread.
>> >> >> > :)
>> >> >> > >
>> >> >> > > Ismael
>> >> >> > >
>> >> >> > > On Tue, Nov 29, 2016 at 4:30 PM, radai
>> >> >> > > 
>> >> >> > wrote:
>> >> >> > >
>> >> >> > > > designing kafka code for stable extensibility is a worthy and
>> >> noble
>> >> >> > > cause.
>> >> >> > > > however, seeing as there are no such derivatives out in the
>> >> >> > > > wild
>> >> >> yet i
>> >> >> > > > think investing the effort right now is a bit premature from
>> >> kafka's
>> >> >> > pov.
>> >> >> > > > I think its enough simply not to purposefully prevent such
>> >> >> extensions.
>> >> >> > > >
>> >> >> > > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma
>> >> >> > > > 
>> >> >> > wrote:
>> >> >> > > >
>> >> >> > > > > On Sat, Nov 26, 2016 at 11:08 PM, radai <
>> >> >> radai.rosenbl...@gmail.com>
>> >> >> > > > > wrote:
>> >> >> > > > >
>> >> >> > > > > > "compatibility guarantees that are expected by people who
>> >> >> subclass
>> >> >> > > > these
>> >> >> > > > > > classes"
>> >> >> > > > > >
>> >> >> > > > > > sorry if this is not the best thread for this discussion,
>> >> >> > > > > > but
>> >> I
>> >> >> > just
>> >> >> > > > > wanted
>> >> >> > > > > > to pop in and say that since any subclassing of these will
>> >> >> > obviously
>> >> >> > > > not
>> >> >> > > > > be
>> >> >> > > > > > done within the kafka codebase - what guarantees are
>> >> >> > > > > > needed?
>> >> >> > > > > >
>> >> >> > > > >
>> >> >> > > > > I elaborated a little in my other message in this thread. A
>> >> simple
>> >> >> > and
>> >> >> > > > > somewhat contrived example: `ConsumerRecord.toString` calls
>> >> >> > > > > the
>> >> >> > `topic`
>> >> >> > > > > method. Someone overrides the `topic` method and it all
>> >> >> > > > > works as
>> >> >> > 

Build failed in Jenkins: kafka-trunk-jdk8 #1064

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-4397: Refactor Connect backing stores for thread safety

--
[...truncated 12181 lines...]
org.apache.kafka.common.record.RecordTest > testChecksum[190] PASSED

org.apache.kafka.common.record.RecordTest > testEquality[190] STARTED

org.apache.kafka.common.record.RecordTest > testEquality[190] PASSED

org.apache.kafka.common.record.RecordTest > testFields[190] STARTED

org.apache.kafka.common.record.RecordTest > testFields[190] PASSED

org.apache.kafka.common.record.RecordTest > testChecksum[191] STARTED

org.apache.kafka.common.record.RecordTest > testChecksum[191] PASSED

org.apache.kafka.common.record.RecordTest > testEquality[191] STARTED

org.apache.kafka.common.record.RecordTest > testEquality[191] PASSED

org.apache.kafka.common.record.RecordTest > testFields[191] STARTED

org.apache.kafka.common.record.RecordTest > testFields[191] PASSED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[0] STARTED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[0] PASSED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[1] STARTED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[1] PASSED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[2] STARTED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[2] PASSED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[3] STARTED

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[3] PASSED

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testSerializationRoundtrip STARTED

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testSerializationRoundtrip PASSED

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testTopiPartitionSerializationCompatibility STARTED

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testTopiPartitionSerializationCompatibility PASSED

org.apache.kafka.common.protocol.ApiKeysTest > testForIdWithInvalidIdLow STARTED

org.apache.kafka.common.protocol.ApiKeysTest > testForIdWithInvalidIdLow PASSED

org.apache.kafka.common.protocol.ApiKeysTest > testForIdWithInvalidIdHigh 
STARTED

org.apache.kafka.common.protocol.ApiKeysTest > testForIdWithInvalidIdHigh PASSED

org.apache.kafka.common.protocol.ErrorsTest > testExceptionName STARTED

org.apache.kafka.common.protocol.ErrorsTest > testExceptionName PASSED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionDefault STARTED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionDefault PASSED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueExceptions STARTED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueExceptions PASSED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionInheritance 
STARTED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionInheritance PASSED

org.apache.kafka.common.protocol.ErrorsTest > testNoneException STARTED

org.apache.kafka.common.protocol.ErrorsTest > testNoneException PASSED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueErrorCodes STARTED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueErrorCodes PASSED

org.apache.kafka.common.protocol.ErrorsTest > testExceptionsAreNotGeneric 
STARTED

org.apache.kafka.common.protocol.ErrorsTest > testExceptionsAreNotGeneric PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testNulls 
STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testNulls 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testToString 
STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testToString 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadStringSizeTooLarge STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadStringSizeTooLarge PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testNullableDefault STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testNullableDefault PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadNegativeStringSize STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadNegativeStringSize PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadArraySizeTooLarge STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadArraySizeTooLarge PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testDefault 
STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testDefault 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadNegativeBytesSize STARTED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testReadNegativeBytesSize PASSED

Re: [DISCUSS] 0.10.1.1 Plan

[Gwen Shapira]

btw. is LinkedIn no longer running from trunk? I'm not used to
LinkedIn employees requesting specific patches to be included in a
bugfix release.

Any discussion on the content of any release is obviously welcome, I'm
just wondering if there was a change in policy.

On Tue, Nov 29, 2016 at 2:17 PM, Ismael Juma  wrote:
> OK, so it seems like there are no changes that break compatibility in the
> 0.10.1 branch since we offer no compatibility guarantees for logging
> output. That's good. :)
>
> About the removal of final, it happened in trunk and it doesn't seem like
> anyone is still asking for it to be included in the 0.10.1 branch so it is
> indeed not important for this bug fix release (I thought we had established
> that quite a while ago).
>
> Ismael
>
> On Tue, Nov 29, 2016 at 9:35 PM, Ignacio Solis  wrote:
>
>> Sorry, that was a hasty reply.  There are also various logging things that
>> change format. This could break parsers.
>>
>> None of them are important, my only argument is that the final keyword
>> removal is not important either.
>>
>> Nacho
>>
>>
>> On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis  wrote:
>>
>> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
>> > 10cfc1628df024f7596d3af5c168fa90f59035ca
>> >
>> > On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma  wrote:
>> >
>> >> Which changes break compatibility in the 0.10.1 branch? It would be good
>> >> to
>> >> fix before the release goes out.
>> >>
>> >> Ismael
>> >>
>> >> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
>> >>
>> >> > Some of the changes in the 0.10.1 branch already are not bug fixes.
>> Some
>> >> > break compatibility.
>> >> >
>> >> > Having said that, at this level we should maintain a stable API and
>> >> leave
>> >> > any changes for real version bumps.  This should be only a bugfix
>> >> release.
>> >> >
>> >> > Nacho
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma 
>> wrote:
>> >> >
>> >> > > I disagree, but let's discuss it another time and in a separate
>> >> thread.
>> >> > :)
>> >> > >
>> >> > > Ismael
>> >> > >
>> >> > > On Tue, Nov 29, 2016 at 4:30 PM, radai 
>> >> > wrote:
>> >> > >
>> >> > > > designing kafka code for stable extensibility is a worthy and
>> noble
>> >> > > cause.
>> >> > > > however, seeing as there are no such derivatives out in the wild
>> >> yet i
>> >> > > > think investing the effort right now is a bit premature from
>> kafka's
>> >> > pov.
>> >> > > > I think its enough simply not to purposefully prevent such
>> >> extensions.
>> >> > > >
>> >> > > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma 
>> >> > wrote:
>> >> > > >
>> >> > > > > On Sat, Nov 26, 2016 at 11:08 PM, radai <
>> >> radai.rosenbl...@gmail.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > "compatibility guarantees that are expected by people who
>> >> subclass
>> >> > > > these
>> >> > > > > > classes"
>> >> > > > > >
>> >> > > > > > sorry if this is not the best thread for this discussion, but
>> I
>> >> > just
>> >> > > > > wanted
>> >> > > > > > to pop in and say that since any subclassing of these will
>> >> > obviously
>> >> > > > not
>> >> > > > > be
>> >> > > > > > done within the kafka codebase - what guarantees are needed?
>> >> > > > > >
>> >> > > > >
>> >> > > > > I elaborated a little in my other message in this thread. A
>> simple
>> >> > and
>> >> > > > > somewhat contrived example: `ConsumerRecord.toString` calls the
>> >> > `topic`
>> >> > > > > method. Someone overrides the `topic` method and it all works as
>> >> > > > expected.
>> >> > > > > In a subsequent release, we change `toString` to use the field
>> >> > directly
>> >> > > > > (like it's done for other fields like `key` and `value`) and it
>> >> will
>> >> > > > break
>> >> > > > > `toString` for this user. One may wonder: why would one
>> override a
>> >> > > method
>> >> > > > > like `topic`? That is a good question, but part of the exercise
>> is
>> >> > > > deciding
>> >> > > > > how we approach these issues. We could make the methods final
>> and
>> >> > > > eliminate
>> >> > > > > the possibility, we could document it so that users can choose
>> to
>> >> do
>> >> > > > weird
>> >> > > > > things if they want, etc.
>> >> > > > >
>> >> > > > > Another thing that is usually good to think about is the
>> >> expectation
>> >> > > for
>> >> > > > > `equals` and `hashCode`. What if subclasses implement them to
>> have
>> >> > > value
>> >> > > > > semantics instead of identity semantics. Is that OK or would it
>> >> break
>> >> > > > > things?
>> >> > > > >
>> >> > > > > Designing for implementation inheritance is generally complex
>> >> > although
>> >> > > > for
>> >> > > > > simple "record" like classes, it can be easier by following a
>> few
>> >> > > > > guidelines.
>> >> > > > >
>> >> > > > > Ismael
>> >> > > > >
>> >> > > >
>> 

[jira] [Commented] (KAFKA-4468) Correctly calculate the window end timestamp after read from state stores

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707119#comment-15707119
 ] 

Bill Bejeck commented on KAFKA-4468:


Picking this one up.

> Correctly calculate the window end timestamp after read from state stores
> -
>
> Key: KAFKA-4468
> URL: https://issues.apache.org/jira/browse/KAFKA-4468
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Bill Bejeck
>  Labels: architecture
>
> When storing the WindowedStore on the persistent KV store, we only use the 
> start timestamp of the window as part of the combo-key as (start-timestamp, 
> key). The reason that we do not add the end-timestamp as well is that we can 
> always calculate it from the start timestamp + window_length, and hence we 
> can save 8 bytes per key on the persistent KV store.
> However, after read it (via {{WindowedDeserializer}}) we do not set its end 
> timestamp correctly but just read it as an {{UnlimitedWindow}}. We should fix 
> this by calculating its end timestamp as mentioned above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4468) Correctly calculate the window end timestamp after read from state stores

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck reassigned KAFKA-4468:
--

Assignee: Bill Bejeck

> Correctly calculate the window end timestamp after read from state stores
> -
>
> Key: KAFKA-4468
> URL: https://issues.apache.org/jira/browse/KAFKA-4468
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Bill Bejeck
>  Labels: architecture
>
> When storing the WindowedStore on the persistent KV store, we only use the 
> start timestamp of the window as part of the combo-key as (start-timestamp, 
> key). The reason that we do not add the end-timestamp as well is that we can 
> always calculate it from the start timestamp + window_length, and hence we 
> can save 8 bytes per key on the persistent KV store.
> However, after read it (via {{WindowedDeserializer}}) we do not set its end 
> timestamp correctly but just read it as an {{UnlimitedWindow}}. We should fix 
> this by calculating its end timestamp as mentioned above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ashish Singh]

On Tue, Nov 29, 2016 at 4:11 PM, Colin McCabe  wrote:

> On Tue, Nov 29, 2016, at 11:39, Ashish Singh wrote:
> > Hello Colin,
> >
> > In the KIP you mentioned that currently the client uses supported api
> > versions information to check if the server supports its desired
> > versions.
> > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > that, but it never went in.
>
> I was taking a look at the RPC implementation today myself and I think
> you're right.  I don't think the client actually invokes
> ApiVersionRequest-- or if it does, I can't find the invocation site.
> Still, the API versions are sent over the wire, and the server does use
> them to support older clients.  I suppose this means that currently,
> clients which are too new fail at the time they attempt to send an RPC
> which is not in the server's supported version range for that RPC.  I
> will edit the KIP to reflect this-- thanks.
>
> > Also, I could not find how you plan to perform
> > version check on client side. In KAFKA-3600, I am maintaining api version
> > for each live connection, and that made a few folks think it is too big
> > of a change.
>
> The version check would be performed by calling ApiVersionRequest, just
> like in your patch.
>
> I think that people are coming around to the idea that we need both
> forward and backwards compatibility for the client.  As the project
> grows, and there are bigger and more deployments, it is probably worth
> spending a little more time on compatibility...
>
> P.S. If you have any spare cycles, it would be great to collaborate on
> this!
>
Kafka-3600 was part of voted-in KIP-35, and a pre-requisite for client
compat. If people are coming around to the idea, maybe we can revisit
KAFKA-3600 and try to find why is it not meaningful to have it in? If we
get KAFKA-3600 in, follow up changes to enable feature selection in clients
should be pretty straight forward.

>
> cheers,
> Colin
>
> >
> > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> > wrote:
> >
> > > Sorry, that link should be:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > >
> > >
> > >
> > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > Hi all,
> > > >
> > > > I've been thinking about a KIP to improve the Kafka client's
> > > > compatibility policy.  If you're interested, please check out:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
>



-- 

Regards,
Ashish

[jira] [Updated] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-4443:

Description: 
Currently in onControllerFailover(), controller will startup 
replicaStatemachine and partitionStateMachine before invoking 
sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq). 
However, if a broker starts right after controller election, the 
LeaderAndIsrRequest sent to follower partitions on this broker will all be 
ignored because broker doesn't know the leaders are alive. 

To fix this problem, in onControllerFailover(), controller should send 
UpdateMetadataRequest to brokers after initializeControllerContext() but before 
it starts replicaStatemachine and partitionStateMachine. The first 
MetadatUpdateRequest will include list of live broker. Although it will not 
include partition leader information, it is OK because we will always send 
MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
replicaStateMachine.startup() and partitionStateMachine.startup().

  was:
Currently in onControllerFailover(), controller will startup 
replicaStatemachine and partitionStateMachine before invoking 
sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq). 
However, if a broker right after controller election, the LeaderAndIsrRequest 
sent to follower partitions on this broker will all be ignored because broker 
doesn't know the leaders are alive. 

To fix this problem, in onControllerFailover(), controller should send 
UpdateMetadataRequest to brokers after initializeControllerContext() but before 
it starts replicaStatemachine and partitionStateMachine. The first 
MetadatUpdateRequest will include list of live broker. Although it will not 
include partition leader information, it is OK because we will always send 
MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
replicaStateMachine.startup() and partitionStateMachine.startup().


> Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest 
> during failover
> -
>
> Key: KAFKA-4443
> URL: https://issues.apache.org/jira/browse/KAFKA-4443
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.1.1
>
>
> Currently in onControllerFailover(), controller will startup 
> replicaStatemachine and partitionStateMachine before invoking 
> sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq).
>  However, if a broker starts right after controller election, the 
> LeaderAndIsrRequest sent to follower partitions on this broker will all be 
> ignored because broker doesn't know the leaders are alive. 
> To fix this problem, in onControllerFailover(), controller should send 
> UpdateMetadataRequest to brokers after initializeControllerContext() but 
> before it starts replicaStatemachine and partitionStateMachine. The first 
> MetadatUpdateRequest will include list of live broker. Although it will not 
> include partition leader information, it is OK because we will always send 
> MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
> replicaStateMachine.startup() and partitionStateMachine.startup().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

On Tue, Nov 29, 2016, at 11:39, Ashish Singh wrote:
> Hello Colin,
> 
> In the KIP you mentioned that currently the client uses supported api
> versions information to check if the server supports its desired
> versions.
> Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> that, but it never went in.

I was taking a look at the RPC implementation today myself and I think
you're right.  I don't think the client actually invokes
ApiVersionRequest-- or if it does, I can't find the invocation site. 
Still, the API versions are sent over the wire, and the server does use
them to support older clients.  I suppose this means that currently,
clients which are too new fail at the time they attempt to send an RPC
which is not in the server's supported version range for that RPC.  I
will edit the KIP to reflect this-- thanks.

> Also, I could not find how you plan to perform
> version check on client side. In KAFKA-3600, I am maintaining api version
> for each live connection, and that made a few folks think it is too big
> of a change.

The version check would be performed by calling ApiVersionRequest, just
like in your patch.

I think that people are coming around to the idea that we need both
forward and backwards compatibility for the client.  As the project
grows, and there are bigger and more deployments, it is probably worth
spending a little more time on compatibility...

P.S. If you have any spare cycles, it would be great to collaborate on
this!

cheers,
Colin

> 
> On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> wrote:
> 
> > Sorry, that link should be:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> >
> >
> >
> > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > Hi all,
> > >
> > > I've been thinking about a KIP to improve the Kafka client's
> > > compatibility policy.  If you're interested, please check out:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Compatibility+Policy
> > >
> > > cheers,
> > > Colin
> >
> 
> 
> 
> -- 
> 
> Regards,
> Ashish

[jira] [Created] (KAFKA-4468) Correctly calculate the window end timestamp after read from state stores

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-4468:


 Summary: Correctly calculate the window end timestamp after read 
from state stores
 Key: KAFKA-4468
 URL: https://issues.apache.org/jira/browse/KAFKA-4468
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang


When storing the WindowedStore on the persistent KV store, we only use the 
start timestamp of the window as part of the combo-key as (start-timestamp, 
key). The reason that we do not add the end-timestamp as well is that we can 
always calculate it from the start timestamp + window_length, and hence we can 
save 8 bytes per key on the persistent KV store.

However, after read it (via {{WindowedDeserializer}}) we do not set its end 
timestamp correctly but just read it as an {{UnlimitedWindow}}. We should fix 
this by calculating its end timestamp as mentioned above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Guozhang Wang]

+1.

On Tue, Nov 29, 2016 at 11:05 AM, Bill Bejeck  wrote:

> +1 (non-binding)
>
> Thanks,
>
> Bill
>
> On Tue, Nov 29, 2016 at 1:34 PM, Matthias J. Sax 
> wrote:
>
> > I’d like to start the voting process for KIP-93:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 93%3A+Improve+invalid+timestamp+handling+in+Kafka+Streams
> >
> > -Matthias
> >
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-94: Session Windows

[Matthias J. Sax]

Very nice KIP!


Couple of comments:

1) Current window API is as follows:

> JoinWindows.of(timeDifference).before(timeDifference).after(timeDifference)
> TimeWindows.of(size).advanceBy(interval)
> UnlimitedWindow.of().startOn(start)

To align with this scheme, I think it would be better to use the
following API for SessionWindows

> SessionWindows.with(inactivityGap)


2) I am wondering, why SessionMerger does need the key?

3) You KIP API for SessionWindows and you PR does not align. There are
some getters in you code that are not part of the KIP (not sure how
important this is)


-Matthias



On 11/24/16 7:59 AM, Damian Guy wrote:
> Hi all,
> 
> I would like to start the discussion on KIP-94:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows
> 
> Thanks,
> Damian
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (KAFKA-4397) Refactor Connect backing stores for thread-safety

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706922#comment-15706922
 ] 

ASF GitHub Bot commented on KAFKA-4397:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2123


> Refactor Connect backing stores for thread-safety
> -
>
> Key: KAFKA-4397
> URL: https://issues.apache.org/jira/browse/KAFKA-4397
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
>Priority: Minor
> Fix For: 0.10.2.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> In Kafka Connect there has been already significant provisioning for 
> multi-threaded execution with respect to classes implementing backing store 
> interfaces. 
> A requirement for 
> [KAFKA-3008|https://issues.apache.org/jira/browse/KAFKA-3008] is to tighten 
> thread-safety guarantees in these implementations, especially for 
> ConfigBackingStore and StatusBackingStore, and this will be the focus of the 
> current ticket. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2123: KAFKA-4397: Refactor Connect backing stores for th...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2123


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4397) Refactor Connect backing stores for thread-safety

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4397.
--
Resolution: Fixed

Issue resolved by pull request 2123
[https://github.com/apache/kafka/pull/2123]

> Refactor Connect backing stores for thread-safety
> -
>
> Key: KAFKA-4397
> URL: https://issues.apache.org/jira/browse/KAFKA-4397
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
>Priority: Minor
> Fix For: 0.10.2.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> In Kafka Connect there has been already significant provisioning for 
> multi-threaded execution with respect to classes implementing backing store 
> interfaces. 
> A requirement for 
> [KAFKA-3008|https://issues.apache.org/jira/browse/KAFKA-3008] is to tighten 
> thread-safety guarantees in these implementations, especially for 
> ConfigBackingStore and StatusBackingStore, and this will be the focus of the 
> current ticket. 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706867#comment-15706867
 ] 

Ismael Juma commented on KAFKA-4443:


The fix version should be 0.10.2.0, right? The PR was only merged to trunk.

> Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest 
> during failover
> -
>
> Key: KAFKA-4443
> URL: https://issues.apache.org/jira/browse/KAFKA-4443
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.1.1
>
>
> Currently in onControllerFailover(), controller will startup 
> replicaStatemachine and partitionStateMachine before invoking 
> sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq).
>  However, if a broker right after controller election, the 
> LeaderAndIsrRequest sent to follower partitions on this broker will all be 
> ignored because broker doesn't know the leaders are alive. 
> To fix this problem, in onControllerFailover(), controller should send 
> UpdateMetadataRequest to brokers after initializeControllerContext() but 
> before it starts replicaStatemachine and partitionStateMachine. The first 
> MetadatUpdateRequest will include list of live broker. Although it will not 
> include partition leader information, it is OK because we will always send 
> MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
> replicaStateMachine.startup() and partitionStateMachine.startup().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin resolved KAFKA-4415.
-
Resolution: Fixed

> Reduce time to create and send MetadataUpdateRequest
> 
>
> Key: KAFKA-4415
> URL: https://issues.apache.org/jira/browse/KAFKA-4415
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.10.1.1
>
>
> As of current implementation, when controller receives 
> ControlledShutdownRequest, it will 1) for every broker in the cluster, for 
> every partition on the broker which wants to shutdown, create an instance of 
> PartitionStateInfo and add it to 
> ControllerBrokerRequestBatch.,updateMetadataRequestMap; and 2) for every 
> broker, for every follower partitions on the broker which wants to shutdown, 
> send one MetadataUpdateRequst to that broker.
> In order to shutdown a broker, the controller will need to instantiate 
> O(partitionNum * brokerNum) PartitionStateInfo and send O(partitionNum * 
> brokerNum) partitionStateInfo. This is not efficient. The broker should only 
> need to instantiate O(partitionNum) PartitionStateInfo and send O(brokerNum) 
> MetadataUpdateRequest.
> Micro-benchmark results show that this optimization can reduce the time of 
> processing ControlledShutdownRequest by 30%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-4415:

Affects Version/s: 0.10.1.0
Fix Version/s: 0.10.1.1

> Reduce time to create and send MetadataUpdateRequest
> 
>
> Key: KAFKA-4415
> URL: https://issues.apache.org/jira/browse/KAFKA-4415
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.10.1.1
>
>
> As of current implementation, when controller receives 
> ControlledShutdownRequest, it will 1) for every broker in the cluster, for 
> every partition on the broker which wants to shutdown, create an instance of 
> PartitionStateInfo and add it to 
> ControllerBrokerRequestBatch.,updateMetadataRequestMap; and 2) for every 
> broker, for every follower partitions on the broker which wants to shutdown, 
> send one MetadataUpdateRequst to that broker.
> In order to shutdown a broker, the controller will need to instantiate 
> O(partitionNum * brokerNum) PartitionStateInfo and send O(partitionNum * 
> brokerNum) partitionStateInfo. This is not efficient. The broker should only 
> need to instantiate O(partitionNum) PartitionStateInfo and send O(brokerNum) 
> MetadataUpdateRequest.
> Micro-benchmark results show that this optimization can reduce the time of 
> processing ControlledShutdownRequest by 30%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-4443:

Affects Version/s: 0.10.1.0
Fix Version/s: 0.10.1.1

> Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest 
> during failover
> -
>
> Key: KAFKA-4443
> URL: https://issues.apache.org/jira/browse/KAFKA-4443
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.1.1
>
>
> Currently in onControllerFailover(), controller will startup 
> replicaStatemachine and partitionStateMachine before invoking 
> sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq).
>  However, if a broker right after controller election, the 
> LeaderAndIsrRequest sent to follower partitions on this broker will all be 
> ignored because broker doesn't know the leaders are alive. 
> To fix this problem, in onControllerFailover(), controller should send 
> UpdateMetadataRequest to brokers after initializeControllerContext() but 
> before it starts replicaStatemachine and partitionStateMachine. The first 
> MetadatUpdateRequest will include list of live broker. Although it will not 
> include partition leader information, it is OK because we will always send 
> MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
> replicaStateMachine.startup() and partitionStateMachine.startup().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4443) Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest during failover

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin resolved KAFKA-4443.
-
Resolution: Fixed

> Controller should send UpdateMetadataRequest prior to LeaderAndIsrRequest 
> during failover
> -
>
> Key: KAFKA-4443
> URL: https://issues.apache.org/jira/browse/KAFKA-4443
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: reliability
> Fix For: 0.10.1.1
>
>
> Currently in onControllerFailover(), controller will startup 
> replicaStatemachine and partitionStateMachine before invoking 
> sendUpdateMetadataRequest(controllerContext.liveOrShuttingDownBrokerIds.toSeq).
>  However, if a broker right after controller election, the 
> LeaderAndIsrRequest sent to follower partitions on this broker will all be 
> ignored because broker doesn't know the leaders are alive. 
> To fix this problem, in onControllerFailover(), controller should send 
> UpdateMetadataRequest to brokers after initializeControllerContext() but 
> before it starts replicaStatemachine and partitionStateMachine. The first 
> MetadatUpdateRequest will include list of live broker. Although it will not 
> include partition leader information, it is OK because we will always send 
> MetadataUpdateRequest again when we send LeaderAndIsrRequest during 
> replicaStateMachine.startup() and partitionStateMachine.startup().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 0.10.1.1 Plan

[Ismael Juma]

OK, so it seems like there are no changes that break compatibility in the
0.10.1 branch since we offer no compatibility guarantees for logging
output. That's good. :)

About the removal of final, it happened in trunk and it doesn't seem like
anyone is still asking for it to be included in the 0.10.1 branch so it is
indeed not important for this bug fix release (I thought we had established
that quite a while ago).

Ismael

On Tue, Nov 29, 2016 at 9:35 PM, Ignacio Solis  wrote:

> Sorry, that was a hasty reply.  There are also various logging things that
> change format. This could break parsers.
>
> None of them are important, my only argument is that the final keyword
> removal is not important either.
>
> Nacho
>
>
> On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis  wrote:
>
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
> > 10cfc1628df024f7596d3af5c168fa90f59035ca
> >
> > On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma  wrote:
> >
> >> Which changes break compatibility in the 0.10.1 branch? It would be good
> >> to
> >> fix before the release goes out.
> >>
> >> Ismael
> >>
> >> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
> >>
> >> > Some of the changes in the 0.10.1 branch already are not bug fixes.
> Some
> >> > break compatibility.
> >> >
> >> > Having said that, at this level we should maintain a stable API and
> >> leave
> >> > any changes for real version bumps.  This should be only a bugfix
> >> release.
> >> >
> >> > Nacho
> >> >
> >> >
> >> >
> >> >
> >> > On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma 
> wrote:
> >> >
> >> > > I disagree, but let's discuss it another time and in a separate
> >> thread.
> >> > :)
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Tue, Nov 29, 2016 at 4:30 PM, radai 
> >> > wrote:
> >> > >
> >> > > > designing kafka code for stable extensibility is a worthy and
> noble
> >> > > cause.
> >> > > > however, seeing as there are no such derivatives out in the wild
> >> yet i
> >> > > > think investing the effort right now is a bit premature from
> kafka's
> >> > pov.
> >> > > > I think its enough simply not to purposefully prevent such
> >> extensions.
> >> > > >
> >> > > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma 
> >> > wrote:
> >> > > >
> >> > > > > On Sat, Nov 26, 2016 at 11:08 PM, radai <
> >> radai.rosenbl...@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > "compatibility guarantees that are expected by people who
> >> subclass
> >> > > > these
> >> > > > > > classes"
> >> > > > > >
> >> > > > > > sorry if this is not the best thread for this discussion, but
> I
> >> > just
> >> > > > > wanted
> >> > > > > > to pop in and say that since any subclassing of these will
> >> > obviously
> >> > > > not
> >> > > > > be
> >> > > > > > done within the kafka codebase - what guarantees are needed?
> >> > > > > >
> >> > > > >
> >> > > > > I elaborated a little in my other message in this thread. A
> simple
> >> > and
> >> > > > > somewhat contrived example: `ConsumerRecord.toString` calls the
> >> > `topic`
> >> > > > > method. Someone overrides the `topic` method and it all works as
> >> > > > expected.
> >> > > > > In a subsequent release, we change `toString` to use the field
> >> > directly
> >> > > > > (like it's done for other fields like `key` and `value`) and it
> >> will
> >> > > > break
> >> > > > > `toString` for this user. One may wonder: why would one
> override a
> >> > > method
> >> > > > > like `topic`? That is a good question, but part of the exercise
> is
> >> > > > deciding
> >> > > > > how we approach these issues. We could make the methods final
> and
> >> > > > eliminate
> >> > > > > the possibility, we could document it so that users can choose
> to
> >> do
> >> > > > weird
> >> > > > > things if they want, etc.
> >> > > > >
> >> > > > > Another thing that is usually good to think about is the
> >> expectation
> >> > > for
> >> > > > > `equals` and `hashCode`. What if subclasses implement them to
> have
> >> > > value
> >> > > > > semantics instead of identity semantics. Is that OK or would it
> >> break
> >> > > > > things?
> >> > > > >
> >> > > > > Designing for implementation inheritance is generally complex
> >> > although
> >> > > > for
> >> > > > > simple "record" like classes, it can be easier by following a
> few
> >> > > > > guidelines.
> >> > > > >
> >> > > > > Ismael
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Nacho - Ignacio Solis - iso...@igso.net
> >> >
> >>
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - iso...@igso.net
> >
>
>
>
> --
> Nacho - Ignacio Solis - iso...@igso.net
>

[jira] [Commented] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706669#comment-15706669
 ] 

ASF GitHub Bot commented on KAFKA-4271:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2189

KAFKA-4271: Fix the server start script for Windows to run normally on a 
32-bit OS

Without this fix the new consumer fails to run on a 32-bit Windows OS.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4271

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2189


commit 1542c2e6d1fe27a07e1cce8c0aefaebd591bed83
Author: Vahid Hashemian 
Date:   2016-11-29T21:46:50Z

KAFKA-4271: Fix the server start script for Windows to run normally on 
32-bit OS too

Without this fix the new consumer fails to run on a 32-bit Windows OS.




> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.
> Update: This issue seems to occur on 32-bit Windows only. I tried this on a 
> Windows 32-bit VM. On a 64-bit machine I did not get the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-4271:
---
Status: Patch Available  (was: Open)

> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.
> Update: This issue seems to occur on 32-bit Windows only. I tried this on a 
> Windows 32-bit VM. On a 64-bit machine I did not get the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-4271:
--

Assignee: Vahid Hashemian

> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.
> Update: This issue seems to occur on 32-bit Windows only. I tried this on a 
> Windows 32-bit VM. On a 64-bit machine I did not get the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2189: KAFKA-4271: Fix the server start script for Window...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/2189

KAFKA-4271: Fix the server start script for Windows to run normally on a 
32-bit OS

Without this fix the new consumer fails to run on a 32-bit Windows OS.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-4271

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2189


commit 1542c2e6d1fe27a07e1cce8c0aefaebd591bed83
Author: Vahid Hashemian 
Date:   2016-11-29T21:46:50Z

KAFKA-4271: Fix the server start script for Windows to run normally on 
32-bit OS too

Without this fix the new consumer fails to run on a 32-bit Windows OS.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] 0.10.1.1 Plan

[Matthias J. Sax]

The commit you mentioned was corrupted and corrected via
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=cc62b4f844ca16eee974e75b736af87b7532de0d

The code change got reverted.

-Matthias

On 11/29/16 1:35 PM, Ignacio Solis wrote:
> Sorry, that was a hasty reply.  There are also various logging things that
> change format. This could break parsers.
> 
> None of them are important, my only argument is that the final keyword
> removal is not important either.
> 
> Nacho
> 
> 
> On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis  wrote:
> 
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
>> 10cfc1628df024f7596d3af5c168fa90f59035ca
>>
>> On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma  wrote:
>>
>>> Which changes break compatibility in the 0.10.1 branch? It would be good
>>> to
>>> fix before the release goes out.
>>>
>>> Ismael
>>>
>>> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
>>>
 Some of the changes in the 0.10.1 branch already are not bug fixes. Some
 break compatibility.

 Having said that, at this level we should maintain a stable API and
>>> leave
 any changes for real version bumps.  This should be only a bugfix
>>> release.

 Nacho




 On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma  wrote:

> I disagree, but let's discuss it another time and in a separate
>>> thread.
 :)
>
> Ismael
>
> On Tue, Nov 29, 2016 at 4:30 PM, radai 
 wrote:
>
>> designing kafka code for stable extensibility is a worthy and noble
> cause.
>> however, seeing as there are no such derivatives out in the wild
>>> yet i
>> think investing the effort right now is a bit premature from kafka's
 pov.
>> I think its enough simply not to purposefully prevent such
>>> extensions.
>>
>> On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma 
 wrote:
>>
>>> On Sat, Nov 26, 2016 at 11:08 PM, radai <
>>> radai.rosenbl...@gmail.com>
>>> wrote:
>>>
 "compatibility guarantees that are expected by people who
>>> subclass
>> these
 classes"

 sorry if this is not the best thread for this discussion, but I
 just
>>> wanted
 to pop in and say that since any subclassing of these will
 obviously
>> not
>>> be
 done within the kafka codebase - what guarantees are needed?

>>>
>>> I elaborated a little in my other message in this thread. A simple
 and
>>> somewhat contrived example: `ConsumerRecord.toString` calls the
 `topic`
>>> method. Someone overrides the `topic` method and it all works as
>> expected.
>>> In a subsequent release, we change `toString` to use the field
 directly
>>> (like it's done for other fields like `key` and `value`) and it
>>> will
>> break
>>> `toString` for this user. One may wonder: why would one override a
> method
>>> like `topic`? That is a good question, but part of the exercise is
>> deciding
>>> how we approach these issues. We could make the methods final and
>> eliminate
>>> the possibility, we could document it so that users can choose to
>>> do
>> weird
>>> things if they want, etc.
>>>
>>> Another thing that is usually good to think about is the
>>> expectation
> for
>>> `equals` and `hashCode`. What if subclasses implement them to have
> value
>>> semantics instead of identity semantics. Is that OK or would it
>>> break
>>> things?
>>>
>>> Designing for implementation inheritance is generally complex
 although
>> for
>>> simple "record" like classes, it can be easier by following a few
>>> guidelines.
>>>
>>> Ismael
>>>
>>
>



 --
 Nacho - Ignacio Solis - iso...@igso.net

>>>
>>
>>
>>
>> --
>> Nacho - Ignacio Solis - iso...@igso.net
>>
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] 0.10.1.1 Plan

[Ignacio Solis]

Sorry, that was a hasty reply.  There are also various logging things that
change format. This could break parsers.

None of them are important, my only argument is that the final keyword
removal is not important either.

Nacho


On Tue, Nov 29, 2016 at 1:25 PM, Ignacio Solis  wrote:

> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=
> 10cfc1628df024f7596d3af5c168fa90f59035ca
>
> On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma  wrote:
>
>> Which changes break compatibility in the 0.10.1 branch? It would be good
>> to
>> fix before the release goes out.
>>
>> Ismael
>>
>> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
>>
>> > Some of the changes in the 0.10.1 branch already are not bug fixes. Some
>> > break compatibility.
>> >
>> > Having said that, at this level we should maintain a stable API and
>> leave
>> > any changes for real version bumps.  This should be only a bugfix
>> release.
>> >
>> > Nacho
>> >
>> >
>> >
>> >
>> > On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma  wrote:
>> >
>> > > I disagree, but let's discuss it another time and in a separate
>> thread.
>> > :)
>> > >
>> > > Ismael
>> > >
>> > > On Tue, Nov 29, 2016 at 4:30 PM, radai 
>> > wrote:
>> > >
>> > > > designing kafka code for stable extensibility is a worthy and noble
>> > > cause.
>> > > > however, seeing as there are no such derivatives out in the wild
>> yet i
>> > > > think investing the effort right now is a bit premature from kafka's
>> > pov.
>> > > > I think its enough simply not to purposefully prevent such
>> extensions.
>> > > >
>> > > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma 
>> > wrote:
>> > > >
>> > > > > On Sat, Nov 26, 2016 at 11:08 PM, radai <
>> radai.rosenbl...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > "compatibility guarantees that are expected by people who
>> subclass
>> > > > these
>> > > > > > classes"
>> > > > > >
>> > > > > > sorry if this is not the best thread for this discussion, but I
>> > just
>> > > > > wanted
>> > > > > > to pop in and say that since any subclassing of these will
>> > obviously
>> > > > not
>> > > > > be
>> > > > > > done within the kafka codebase - what guarantees are needed?
>> > > > > >
>> > > > >
>> > > > > I elaborated a little in my other message in this thread. A simple
>> > and
>> > > > > somewhat contrived example: `ConsumerRecord.toString` calls the
>> > `topic`
>> > > > > method. Someone overrides the `topic` method and it all works as
>> > > > expected.
>> > > > > In a subsequent release, we change `toString` to use the field
>> > directly
>> > > > > (like it's done for other fields like `key` and `value`) and it
>> will
>> > > > break
>> > > > > `toString` for this user. One may wonder: why would one override a
>> > > method
>> > > > > like `topic`? That is a good question, but part of the exercise is
>> > > > deciding
>> > > > > how we approach these issues. We could make the methods final and
>> > > > eliminate
>> > > > > the possibility, we could document it so that users can choose to
>> do
>> > > > weird
>> > > > > things if they want, etc.
>> > > > >
>> > > > > Another thing that is usually good to think about is the
>> expectation
>> > > for
>> > > > > `equals` and `hashCode`. What if subclasses implement them to have
>> > > value
>> > > > > semantics instead of identity semantics. Is that OK or would it
>> break
>> > > > > things?
>> > > > >
>> > > > > Designing for implementation inheritance is generally complex
>> > although
>> > > > for
>> > > > > simple "record" like classes, it can be easier by following a few
>> > > > > guidelines.
>> > > > >
>> > > > > Ismael
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Nacho - Ignacio Solis - iso...@igso.net
>> >
>>
>
>
>
> --
> Nacho - Ignacio Solis - iso...@igso.net
>



-- 
Nacho - Ignacio Solis - iso...@igso.net

Re: [DISCUSS] 0.10.1.1 Plan

[Ignacio Solis]

https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=commit;h=10cfc1628df024f7596d3af5c168fa90f59035ca

On Tue, Nov 29, 2016 at 1:24 PM, Ismael Juma  wrote:

> Which changes break compatibility in the 0.10.1 branch? It would be good to
> fix before the release goes out.
>
> Ismael
>
> On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:
>
> > Some of the changes in the 0.10.1 branch already are not bug fixes. Some
> > break compatibility.
> >
> > Having said that, at this level we should maintain a stable API and leave
> > any changes for real version bumps.  This should be only a bugfix
> release.
> >
> > Nacho
> >
> >
> >
> >
> > On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma  wrote:
> >
> > > I disagree, but let's discuss it another time and in a separate thread.
> > :)
> > >
> > > Ismael
> > >
> > > On Tue, Nov 29, 2016 at 4:30 PM, radai 
> > wrote:
> > >
> > > > designing kafka code for stable extensibility is a worthy and noble
> > > cause.
> > > > however, seeing as there are no such derivatives out in the wild yet
> i
> > > > think investing the effort right now is a bit premature from kafka's
> > pov.
> > > > I think its enough simply not to purposefully prevent such
> extensions.
> > > >
> > > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > On Sat, Nov 26, 2016 at 11:08 PM, radai <
> radai.rosenbl...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > "compatibility guarantees that are expected by people who
> subclass
> > > > these
> > > > > > classes"
> > > > > >
> > > > > > sorry if this is not the best thread for this discussion, but I
> > just
> > > > > wanted
> > > > > > to pop in and say that since any subclassing of these will
> > obviously
> > > > not
> > > > > be
> > > > > > done within the kafka codebase - what guarantees are needed?
> > > > > >
> > > > >
> > > > > I elaborated a little in my other message in this thread. A simple
> > and
> > > > > somewhat contrived example: `ConsumerRecord.toString` calls the
> > `topic`
> > > > > method. Someone overrides the `topic` method and it all works as
> > > > expected.
> > > > > In a subsequent release, we change `toString` to use the field
> > directly
> > > > > (like it's done for other fields like `key` and `value`) and it
> will
> > > > break
> > > > > `toString` for this user. One may wonder: why would one override a
> > > method
> > > > > like `topic`? That is a good question, but part of the exercise is
> > > > deciding
> > > > > how we approach these issues. We could make the methods final and
> > > > eliminate
> > > > > the possibility, we could document it so that users can choose to
> do
> > > > weird
> > > > > things if they want, etc.
> > > > >
> > > > > Another thing that is usually good to think about is the
> expectation
> > > for
> > > > > `equals` and `hashCode`. What if subclasses implement them to have
> > > value
> > > > > semantics instead of identity semantics. Is that OK or would it
> break
> > > > > things?
> > > > >
> > > > > Designing for implementation inheritance is generally complex
> > although
> > > > for
> > > > > simple "record" like classes, it can be easier by following a few
> > > > > guidelines.
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - iso...@igso.net
> >
>



-- 
Nacho - Ignacio Solis - iso...@igso.net

Re: [DISCUSS] 0.10.1.1 Plan

[Ismael Juma]

Which changes break compatibility in the 0.10.1 branch? It would be good to
fix before the release goes out.

Ismael

On 29 Nov 2016 9:09 pm, "Ignacio Solis"  wrote:

> Some of the changes in the 0.10.1 branch already are not bug fixes. Some
> break compatibility.
>
> Having said that, at this level we should maintain a stable API and leave
> any changes for real version bumps.  This should be only a bugfix release.
>
> Nacho
>
>
>
>
> On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma  wrote:
>
> > I disagree, but let's discuss it another time and in a separate thread.
> :)
> >
> > Ismael
> >
> > On Tue, Nov 29, 2016 at 4:30 PM, radai 
> wrote:
> >
> > > designing kafka code for stable extensibility is a worthy and noble
> > cause.
> > > however, seeing as there are no such derivatives out in the wild yet i
> > > think investing the effort right now is a bit premature from kafka's
> pov.
> > > I think its enough simply not to purposefully prevent such extensions.
> > >
> > > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma 
> wrote:
> > >
> > > > On Sat, Nov 26, 2016 at 11:08 PM, radai 
> > > > wrote:
> > > >
> > > > > "compatibility guarantees that are expected by people who subclass
> > > these
> > > > > classes"
> > > > >
> > > > > sorry if this is not the best thread for this discussion, but I
> just
> > > > wanted
> > > > > to pop in and say that since any subclassing of these will
> obviously
> > > not
> > > > be
> > > > > done within the kafka codebase - what guarantees are needed?
> > > > >
> > > >
> > > > I elaborated a little in my other message in this thread. A simple
> and
> > > > somewhat contrived example: `ConsumerRecord.toString` calls the
> `topic`
> > > > method. Someone overrides the `topic` method and it all works as
> > > expected.
> > > > In a subsequent release, we change `toString` to use the field
> directly
> > > > (like it's done for other fields like `key` and `value`) and it will
> > > break
> > > > `toString` for this user. One may wonder: why would one override a
> > method
> > > > like `topic`? That is a good question, but part of the exercise is
> > > deciding
> > > > how we approach these issues. We could make the methods final and
> > > eliminate
> > > > the possibility, we could document it so that users can choose to do
> > > weird
> > > > things if they want, etc.
> > > >
> > > > Another thing that is usually good to think about is the expectation
> > for
> > > > `equals` and `hashCode`. What if subclasses implement them to have
> > value
> > > > semantics instead of identity semantics. Is that OK or would it break
> > > > things?
> > > >
> > > > Designing for implementation inheritance is generally complex
> although
> > > for
> > > > simple "record" like classes, it can be easier by following a few
> > > > guidelines.
> > > >
> > > > Ismael
> > > >
> > >
> >
>
>
>
> --
> Nacho - Ignacio Solis - iso...@igso.net
>

Re: [DISCUSS] 0.10.1.1 Plan

[Ignacio Solis]

Some of the changes in the 0.10.1 branch already are not bug fixes. Some
break compatibility.

Having said that, at this level we should maintain a stable API and leave
any changes for real version bumps.  This should be only a bugfix release.

Nacho




On Tue, Nov 29, 2016 at 8:35 AM, Ismael Juma  wrote:

> I disagree, but let's discuss it another time and in a separate thread. :)
>
> Ismael
>
> On Tue, Nov 29, 2016 at 4:30 PM, radai  wrote:
>
> > designing kafka code for stable extensibility is a worthy and noble
> cause.
> > however, seeing as there are no such derivatives out in the wild yet i
> > think investing the effort right now is a bit premature from kafka's pov.
> > I think its enough simply not to purposefully prevent such extensions.
> >
> > On Tue, Nov 29, 2016 at 4:05 AM, Ismael Juma  wrote:
> >
> > > On Sat, Nov 26, 2016 at 11:08 PM, radai 
> > > wrote:
> > >
> > > > "compatibility guarantees that are expected by people who subclass
> > these
> > > > classes"
> > > >
> > > > sorry if this is not the best thread for this discussion, but I just
> > > wanted
> > > > to pop in and say that since any subclassing of these will obviously
> > not
> > > be
> > > > done within the kafka codebase - what guarantees are needed?
> > > >
> > >
> > > I elaborated a little in my other message in this thread. A simple and
> > > somewhat contrived example: `ConsumerRecord.toString` calls the `topic`
> > > method. Someone overrides the `topic` method and it all works as
> > expected.
> > > In a subsequent release, we change `toString` to use the field directly
> > > (like it's done for other fields like `key` and `value`) and it will
> > break
> > > `toString` for this user. One may wonder: why would one override a
> method
> > > like `topic`? That is a good question, but part of the exercise is
> > deciding
> > > how we approach these issues. We could make the methods final and
> > eliminate
> > > the possibility, we could document it so that users can choose to do
> > weird
> > > things if they want, etc.
> > >
> > > Another thing that is usually good to think about is the expectation
> for
> > > `equals` and `hashCode`. What if subclasses implement them to have
> value
> > > semantics instead of identity semantics. Is that OK or would it break
> > > things?
> > >
> > > Designing for implementation inheritance is generally complex although
> > for
> > > simple "record" like classes, it can be easier by following a few
> > > guidelines.
> > >
> > > Ismael
> > >
> >
>



-- 
Nacho - Ignacio Solis - iso...@igso.net

[jira] [Assigned] (KAFKA-4458) Add per partition metrics for in-sync and assigned replica count

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xavier Léauté reassigned KAFKA-4458:


Assignee: Xavier Léauté

> Add per partition metrics for in-sync and assigned replica count
> 
>
> Key: KAFKA-4458
> URL: https://issues.apache.org/jira/browse/KAFKA-4458
> Project: Kafka
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count
>  for details on proposed changes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #1063

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-4427: Skip topic groups with no tasks

[me] MINOR: Make release notes script check resolutions to avoid spurious

[becket.qin] KAFKA-4415; Reduce time to create and send UpdateMetadataRequest

--
[...truncated 3892 lines...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > 

[jira] [Commented] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706513#comment-15706513
 ] 

Vahid Hashemian commented on KAFKA-4271:


[~yuyan] Thanks for the pointer. I can confirm that your suggested setting 
worked for me.

> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.
> Update: This issue seems to occur on 32-bit Windows only. I tried this on a 
> Windows 32-bit VM. On a 64-bit machine I did not get the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Ignacio Solis]

I'm ok with 32 bit keys and leaving the interpretation out of this
discussion/KIP.

Nacho

On Tue, Nov 29, 2016 at 3:35 AM, Michael Pearce 
wrote:

> I assume, that after a period of a week, that there is no concerns now
> with points 1, and 2 and now we have agreement that headers are useful and
> needed in Kafka. As such if put to a KIP vote, this wouldn’t be a reason to
> reject.
>
> @
> Ignacio on point 4).
> I think for purpose of getting this KIP moving past this, we can state the
> key will be a 4 bytes space that can will be naturally interpreted as an
> Int32 (if namespacing is later wanted you can easily split this into two
> int16 spaces), from the wire protocol implementation this makes no
> difference I don’t believe. Is this reasonable to all?
>
> On 5) as per point 4 therefor happy we keep with 32 bits.
>
>
>
>
>
>
> On 18/11/2016, 20:34, "ignacio.so...@gmail.com on behalf of Ignacio
> Solis"  wrote:
>
> Summary:
>
> 3) Yes - Header value as byte[]
>
> 4a) Int,Int - No
> 4b) Int - Yes
> 4c) String - Reluctant maybe
>
> 5) I believe the header system should take a single int.  I think
> 32bits is
> a good size, if you want to interpret this as to 16bit numbers in the
> layer
> above go right ahead.  If somebody wants to argue for 16 bits or 64
> bits of
> header key space I would listen.
>
>
> Discussion:
> Dividing the key space into sub_key_1 and sub_key_2 makes no sense to
> me at
> this layer.  Are we going to start providing APIs to get all the
> sub_key_1s? or all the sub_key_2s?  If there is no distinguishing
> functions
> that are applied to each one then they should be a single value.  At
> this
> layer all we're doing is equality.
> If the above layer wants to interpret this as 2, 3 or more values
> that's a
> different question.  I personally think it's all one keyspace that is
> getting assigned using some structure, but if you want to sub-assign
> parts
> of it then that's fine.
>
> The same discussion applies to strings.  If somebody argued for
> strings,
> would we be arguing to divide the strings with dots ('.') as a
> requirement?
> Would we want them to give us the different name segments separately?
> Would we be performing any actions on this key other than matching?
>
> Nacho
>
>
>
> On Fri, Nov 18, 2016 at 9:30 AM, Michael Pearce  >
> wrote:
>
> > #jay #jun any concerns on 1 and 2 still?
> >
> > @all
> > To get this moving along a bit more I'd also like to ask to get
> clarity on
> > the below last points:
> >
> > 3) I believe we're all roughly happy with the header value being a
> byte[]?
> >
> > 4) I believe consensus has been for an namespace based int approach
> > {int,int} for the key. Any objections if this is what we go with?
> >
> > 5) as we have if assumption in (4)  is correct, {int,int} keys.
> > Should both int's be int16 or int32?
> > I'm for them being int16(2 bytes) as combined is space of 4bytes as
> per
> > original and gives plenty of combinations for the foreseeable, and
> keeps
> > the overhead small.
> >
> > Do we see any benefit in another kip call to discuss these at all?
> >
> > Cheers
> > Mike
> > 
> > From: K Burstev 
> > Sent: Friday, November 18, 2016 7:07:07 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > For what it is worth also i agree. As a user:
> >
> >  1) Yes - Headers are worthwhile
> >  2) Yes - Headers should be a top level option
> >
> > 14.11.2016, 21:15, "Ignacio Solis" :
> > > 1) Yes - Headers are worthwhile
> > > 2) Yes - Headers should be a top level option
> > >
> > > On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce <
> michael.pea...@ig.com>
> > > wrote:
> > >
> > >>  Hi Roger,
> > >>
> > >>  The kip details/examples the original proposal for key spacing ,
> not
> > the
> > >>  new mentioned as per discussion namespace idea.
> > >>
> > >>  We will need to update the kip, when we get agreement this is a
> better
> > >>  approach (which seems to be the case if I have understood the
> general
> > >>  feeling in the conversation)
> > >>
> > >>  Re the variable ints, at very early stage we did think about
> this. I
> > think
> > >>  the added complexity for the saving isn't worth it. I'd rather go
> > with, if
> > >>  we want to reduce overheads and size int16 (2bytes) keys as it
> keeps it
> > >>  simple.
> > >>
> > >>  On the note of no headers, there is as per the kip as we use an
> > attribute
> > >>  bit to denote if headers are present or not as such provides a
> zero

[jira] [Commented] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706416#comment-15706416
 ] 

ASF GitHub Bot commented on KAFKA-4415:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2169


> Reduce time to create and send MetadataUpdateRequest
> 
>
> Key: KAFKA-4415
> URL: https://issues.apache.org/jira/browse/KAFKA-4415
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> As of current implementation, when controller receives 
> ControlledShutdownRequest, it will 1) for every broker in the cluster, for 
> every partition on the broker which wants to shutdown, create an instance of 
> PartitionStateInfo and add it to 
> ControllerBrokerRequestBatch.,updateMetadataRequestMap; and 2) for every 
> broker, for every follower partitions on the broker which wants to shutdown, 
> send one MetadataUpdateRequst to that broker.
> In order to shutdown a broker, the controller will need to instantiate 
> O(partitionNum * brokerNum) PartitionStateInfo and send O(partitionNum * 
> brokerNum) partitionStateInfo. This is not efficient. The broker should only 
> need to instantiate O(partitionNum) PartitionStateInfo and send O(brokerNum) 
> MetadataUpdateRequest.
> Micro-benchmark results show that this optimization can reduce the time of 
> processing ControlledShutdownRequest by 30%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2169: KAFKA-4415; Reduce time to create and send Metadat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2169


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3123) Follower Broker cannot start if offsets are already out of range

[Soumyajit Sahu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706415#comment-15706415
 ] 

Soumyajit Sahu commented on KAFKA-3123:
---

I am closing this previously opened (and now obsolete) PR.

> Follower Broker cannot start if offsets are already out of range
> 
>
> Key: KAFKA-3123
> URL: https://issues.apache.org/jira/browse/KAFKA-3123
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Affects Versions: 0.9.0.0
>Reporter: Soumyajit Sahu
>Assignee: Soumyajit Sahu
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.2.0
>
> Attachments: 
> 0001-Fix-Follower-crashes-when-offset-out-of-range-during.patch
>
>
> I was trying to upgrade our test Windows cluster from 0.8.1.1 to 0.9.0 one 
> machine at a time. Our logs have just 2 hours of retention. I had re-imaged 
> the test machine under consideration, and got the following error in loop 
> after starting afresh with 0.9.0 broker:
> [2016-01-19 13:57:28,809] WARN [ReplicaFetcherThread-1-169595708], Replica 
> 15588 for partition [EventLogs4,1] reset its fetch offset from 0 to 
> current leader 169595708's start offset 334086 
> (kafka.server.ReplicaFetcherThread)
> [2016-01-19 13:57:28,809] ERROR [ReplicaFetcherThread-1-169595708], Error 
> getting offset for partition [EventLogs4,1] to broker 169595708 
> (kafka.server.ReplicaFetcherThread)
> java.lang.IllegalStateException: Compaction for partition [EXO_EventLogs4,1] 
> cannot be aborted and paused since it is in LogCleaningPaused state.
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply$mcV$sp(LogCleanerManager.scala:149)
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply(LogCleanerManager.scala:140)
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply(LogCleanerManager.scala:140)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at 
> kafka.log.LogCleanerManager.abortAndPauseCleaning(LogCleanerManager.scala:140)
>   at kafka.log.LogCleaner.abortAndPauseCleaning(LogCleaner.scala:141)
>   at kafka.log.LogManager.truncateFullyAndStartAt(LogManager.scala:304)
>   at 
> kafka.server.ReplicaFetcherThread.handleOffsetOutOfRange(ReplicaFetcherThread.scala:185)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:152)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:122)
>   at scala.Option.foreach(Option.scala:236)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:122)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:120)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:120)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:120)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:120)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:93)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> I could unblock myself with a code change. I deleted the action for "case s 
> =>" in the LogCleanerManager.scala's abortAndPauseCleaning(). I think we 
> should not throw exception if the state is already LogCleaningAborted or 
> LogCleaningPaused in this function, but instead just let it roll.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3123) Follower Broker cannot start if offsets are already out of range

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706410#comment-15706410
 ] 

ASF GitHub Bot commented on KAFKA-3123:
---

Github user soumyajit-sahu closed the pull request at:

https://github.com/apache/kafka/pull/1028


> Follower Broker cannot start if offsets are already out of range
> 
>
> Key: KAFKA-3123
> URL: https://issues.apache.org/jira/browse/KAFKA-3123
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Affects Versions: 0.9.0.0
>Reporter: Soumyajit Sahu
>Assignee: Soumyajit Sahu
>Priority: Critical
>  Labels: patch
> Fix For: 0.10.2.0
>
> Attachments: 
> 0001-Fix-Follower-crashes-when-offset-out-of-range-during.patch
>
>
> I was trying to upgrade our test Windows cluster from 0.8.1.1 to 0.9.0 one 
> machine at a time. Our logs have just 2 hours of retention. I had re-imaged 
> the test machine under consideration, and got the following error in loop 
> after starting afresh with 0.9.0 broker:
> [2016-01-19 13:57:28,809] WARN [ReplicaFetcherThread-1-169595708], Replica 
> 15588 for partition [EventLogs4,1] reset its fetch offset from 0 to 
> current leader 169595708's start offset 334086 
> (kafka.server.ReplicaFetcherThread)
> [2016-01-19 13:57:28,809] ERROR [ReplicaFetcherThread-1-169595708], Error 
> getting offset for partition [EventLogs4,1] to broker 169595708 
> (kafka.server.ReplicaFetcherThread)
> java.lang.IllegalStateException: Compaction for partition [EXO_EventLogs4,1] 
> cannot be aborted and paused since it is in LogCleaningPaused state.
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply$mcV$sp(LogCleanerManager.scala:149)
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply(LogCleanerManager.scala:140)
>   at 
> kafka.log.LogCleanerManager$$anonfun$abortAndPauseCleaning$1.apply(LogCleanerManager.scala:140)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at 
> kafka.log.LogCleanerManager.abortAndPauseCleaning(LogCleanerManager.scala:140)
>   at kafka.log.LogCleaner.abortAndPauseCleaning(LogCleaner.scala:141)
>   at kafka.log.LogManager.truncateFullyAndStartAt(LogManager.scala:304)
>   at 
> kafka.server.ReplicaFetcherThread.handleOffsetOutOfRange(ReplicaFetcherThread.scala:185)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:152)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$2.apply(AbstractFetcherThread.scala:122)
>   at scala.Option.foreach(Option.scala:236)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:122)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:120)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:120)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:120)
>   at 
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:120)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:93)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> I could unblock myself with a code change. I deleted the action for "case s 
> =>" in the LogCleanerManager.scala's abortAndPauseCleaning(). I think we 
> should not throw exception if the state is already LogCleaningAborted or 
> LogCleaningPaused in this function, but instead just let it roll.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1028: KAFKA-3123: Follower Broker cannot start if offset...

[soumyajit-sahu]

Github user soumyajit-sahu closed the pull request at:

https://github.com/apache/kafka/pull/1028


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ashish Singh]

Hello Colin,

In the KIP you mentioned that currently the client uses supported api
versions information to check if the server supports its desired versions.
Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
that, but it never went in. Also, I could not find how you plan to perform
version check on client side. In KAFKA-3600, I am maintaining api version
for each live connection, and that made a few folks think it is too big of
a change.

On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe  wrote:

> Sorry, that link should be:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>
>
>
> On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > Hi all,
> >
> > I've been thinking about a KIP to improve the Kafka client's
> > compatibility policy.  If you're interested, please check out:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 97%3A+Improved+Kafka+Compatibility+Policy
> >
> > cheers,
> > Colin
>



-- 

Regards,
Ashish

[GitHub] kafka pull request #2174: MINOR: Make release notes script check resolutions...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2174


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4467) Run tests on travis-ci using docker

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created KAFKA-4467:
--

 Summary: Run tests on travis-ci using docker
 Key: KAFKA-4467
 URL: https://issues.apache.org/jira/browse/KAFKA-4467
 Project: Kafka
  Issue Type: Sub-task
Reporter: Raghav Kumar Gautam






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4466) Add support to ducktape to run only a part of all tests

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created KAFKA-4466:
--

 Summary: Add support to ducktape to run only a part of all tests
 Key: KAFKA-4466
 URL: https://issues.apache.org/jira/browse/KAFKA-4466
 Project: Kafka
  Issue Type: Sub-task
Reporter: Raghav Kumar Gautam






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4464) Clean shutdown of broker fails due to controller error

[Xavier Lange (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706283#comment-15706283
 ] 

Xavier Lange commented on KAFKA-4464:
-

Here is my kafka broker config:

{code}
kafka@86a156fd9dda:~$ cat /kafka/config/server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults

# Server Basics #

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
auto.leader.rebalance.enable=true

# Replication
auto.create.topics.enable=true
default.replication.factor=2

# Hostname the broker will advertise to consumers. If not set, kafka will use 
the value returned
# from InetAddress.getLocalHost().  If there are multiple interfaces 
getLocalHost
# may not be what you want.
advertised.host.name=10.60.68.122

# Enable topic deletion
delete.topic.enable=true

# Socket Server Settings 
#

# The port the socket server listens on
port=9092
advertised.port=9092

num.io.threads=8
num.network.threads=8
socket.request.max.bytes=104857600
socket.receive.buffer.bytes=1048576
socket.send.buffer.bytes=1048576
queued.max.requests=16
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100

# Replication Settings #

num.replica.fetchers=4

# Log Basics #

# The directory under which to store log files
log.dir=/data
log.dirs=/data

# The number of logical partitions per topic per server. More partitions allow 
greater parallelism
# for consumption, but also mean more files.
num.partitions=20
num.network.threads=20

# Log Retention Policy #

# The following configurations control the disposal of log segments. The policy 
can
# be set to delete segments after a period of time, or after a given size has 
accumulated.
# A segment will be deleted whenever *either* of these criteria are met. 
Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion
# log.retention.hours=168
# 10 years
log.retention.hours=87600

# Zookeeper #

# Zk connection string (see zk docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=itsecmon-zk1.usw1.viasat.cloud:2181,itsecmon-zk2.usw1.viasat.cloud:2181,itsecmon-zk3.usw1.viasat.cloud:2181,itsecmon-zk4.usw1.viasat.cloud:2181,itsecmon-zk5.usw1.viasat.cloud:2181
zookeeper.connection.timeout.ms=1
controlled.shutdown.enable=true
zookeeper.session.timeout.ms=1

# vim:set filetype=jproperties
{code}

> Clean shutdown of broker fails due to controller error
> --
>
> Key: KAFKA-4464
> URL: https://issues.apache.org/jira/browse/KAFKA-4464
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.10.1.0
> Environment: kafka@86a156fd9dda:~$ java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> kafka@86a156fd9dda:~$ uname -a
> Linux 86a156fd9dda 4.7.3-coreos-r2 #1 SMP Tue Nov 1 01:38:43 UTC 2016 x86_64 
> x86_64 x86_64 GNU/Linux
> kafka@86a156fd9dda:~$ ps alx | grep java
> 4  1000 1 0  20   0 75887304 3820220 futex_ Ssl ? 9379:49 
> /usr/lib/jvm/java-8-oracle/bin/java -Xmx3G -Xms3G -server -XX:+UseG1GC 
> -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 
> -XX:+DisableExplicitGC -Djava.awt.headless=true 
> -Xloggc:/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails 
> -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
> -Dcom.sun.management.jmxremote=true 
> 

[jira] [Created] (KAFKA-4465) Create docker image and scripts for running tests locally

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created KAFKA-4465:
--

 Summary: Create docker image and scripts for running tests locally
 Key: KAFKA-4465
 URL: https://issues.apache.org/jira/browse/KAFKA-4465
 Project: Kafka
  Issue Type: Sub-task
Reporter: Raghav Kumar Gautam
Assignee: Raghav Kumar Gautam






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4462) Improved Kafka Client Compatibility Policy

[Colin P. McCabe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe updated KAFKA-4462:
---
Description: A proposal to improve the compatibility policy of the Kafka 
client by supporting the combination of new client, old broker.  See 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
 for more details.  (was: A proposal to improve the compatibility policy of the 
Kafka client by supporting the combination of new client, old broker.  See 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
 for more details.)

> Improved Kafka Client Compatibility Policy
> --
>
> Key: KAFKA-4462
> URL: https://issues.apache.org/jira/browse/KAFKA-4462
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.1.1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>
> A proposal to improve the compatibility policy of the Kafka client by 
> supporting the combination of new client, old broker.  See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4464) Clean shutdown of broker fails due to controller error

[Xavier Lange (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xavier Lange updated KAFKA-4464:

Affects Version/s: 0.10.1.0

> Clean shutdown of broker fails due to controller error
> --
>
> Key: KAFKA-4464
> URL: https://issues.apache.org/jira/browse/KAFKA-4464
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.10.1.0
> Environment: kafka@86a156fd9dda:~$ java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> kafka@86a156fd9dda:~$ uname -a
> Linux 86a156fd9dda 4.7.3-coreos-r2 #1 SMP Tue Nov 1 01:38:43 UTC 2016 x86_64 
> x86_64 x86_64 GNU/Linux
> kafka@86a156fd9dda:~$ ps alx | grep java
> 4  1000 1 0  20   0 75887304 3820220 futex_ Ssl ? 9379:49 
> /usr/lib/jvm/java-8-oracle/bin/java -Xmx3G -Xms3G -server -XX:+UseG1GC 
> -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 
> -XX:+DisableExplicitGC -Djava.awt.headless=true 
> -Xloggc:/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails 
> -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
> -Dcom.sun.management.jmxremote=true 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Djava.rmi.server.hostname=10.60.68.122 
> -Dcom.sun.management.jmxremote.rmi.port=7203 -Djava.net.preferIPv4Stack=true 
> -Dcom.sun.management.jmxremote.port=7203 -Dkafka.logs.dir=/kafka/bin/../logs 
> -Dlog4j.configuration=file:/kafka/bin/../config/log4j.properties -cp 
> :/kafka/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/kafka/bin/../libs/argparse4j-0.5.0.jar:/kafka/bin/../libs/connect-api-0.10.1.0.jar:/kafka/bin/../libs/connect-file-0.10.1.0.jar:/kafka/bin/../libs/connect-json-0.10.1.0.jar:/kafka/bin/../libs/connect-runtime-0.10.1.0.jar:/kafka/bin/../libs/guava-18.0.jar:/kafka/bin/../libs/hk2-api-2.4.0-b34.jar:/kafka/bin/../libs/hk2-locator-2.4.0-b34.jar:/kafka/bin/../libs/hk2-utils-2.4.0-b34.jar:/kafka/bin/../libs/jackson-annotations-2.6.0.jar:/kafka/bin/../libs/jackson-core-2.6.3.jar:/kafka/bin/../libs/jackson-databind-2.6.3.jar:/kafka/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/kafka/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/kafka/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/kafka/bin/../libs/javassist-3.18.2-GA.jar:/kafka/bin/../libs/javax.annotation-api-1.2.jar:/kafka/bin/../libs/javax.inject-1.jar:/kafka/bin/../libs/javax.inject-2.4.0-b34.jar:/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/kafka/bin/../libs/jersey-client-2.22.2.jar:/kafka/bin/../libs/jersey-common-2.22.2.jar:/kafka/bin/../libs/jersey-container-servlet-2.22.2.jar:/kafka/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/kafka/bin/../libs/jersey-guava-2.22.2.jar:/kafka/bin/../libs/jersey-media-jaxb-2.22.2.jar:/kafka/bin/../libs/jersey-server-2.22.2.jar:/kafka/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-http-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-io-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-security-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-server-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-util-9.2.15.v20160210.jar:/kafka/bin/../libs/jopt-simple-4.9.jar:/kafka/bin/../libs/kafka-clients-0.10.1.0.jar:/kafka/bin/../libs/kafka-log4j-appender-0.10.1.0.jar:/kafka/bin/../libs/kafka-streams-0.10.1.0.jar:/kafka/bin/../libs/kafka-streams-examples-0.10.1.0.jar:/kafka/bin/../libs/kafka-tools-0.10.1.0.jar:/kafka/bin/../libs/kafka_2.11-0.10.1.0-sources.jar:/kafka/bin/../libs/kafka_2.11-0.10.1.0-test-sources.jar:/kafka/bin/../libs/kafka_2.11-0.10.1.0.jar:/kafka/bin/../libs/log4j-1.2.17.jar:/kafka/bin/../libs/lz4-1.3.0.jar:/kafka/bin/../libs/metrics-core-2.2.0.jar:/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/kafka/bin/../libs/reflections-0.9.10.jar:/kafka/bin/../libs/rocksdbjni-4.9.0.jar:/kafka/bin/../libs/scala-library-2.11.8.jar:/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/kafka/bin/../libs/slf4j-api-1.7.21.jar:/kafka/bin/../libs/slf4j-log4j12-1.7.21.jar:/kafka/bin/../libs/snappy-java-1.1.2.6.jar:/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/kafka/bin/../libs/zkclient-0.9.jar:/kafka/bin/../libs/zookeeper-3.4.8.jar
>  kafka.Kafka /kafka/config/server.properties
> This is running inside a docker container.
>Reporter: Xavier Lange
>
> My cluster is unable to communicate to one of my brokers (Broker 1 in this 
> case) and is spinning on logs:
> {code}
> [2016-11-29 19:05:08,659] WARN [ReplicaFetcherThread-0-1], Error in fetch 
> kafka.server.ReplicaFetcherThread$FetchRequest@27aeb5f4 
> (kafka.server.ReplicaFetcherThread)

[jira] [Created] (KAFKA-4464) Clean shutdown of broker fails due to controller error

[Xavier Lange (JIRA)]

Xavier Lange created KAFKA-4464:
---

 Summary: Clean shutdown of broker fails due to controller error
 Key: KAFKA-4464
 URL: https://issues.apache.org/jira/browse/KAFKA-4464
 Project: Kafka
  Issue Type: Bug
  Components: controller
 Environment: kafka@86a156fd9dda:~$ java -version
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

kafka@86a156fd9dda:~$ uname -a
Linux 86a156fd9dda 4.7.3-coreos-r2 #1 SMP Tue Nov 1 01:38:43 UTC 2016 x86_64 
x86_64 x86_64 GNU/Linux

kafka@86a156fd9dda:~$ ps alx | grep java
4  1000 1 0  20   0 75887304 3820220 futex_ Ssl ? 9379:49 
/usr/lib/jvm/java-8-oracle/bin/java -Xmx3G -Xms3G -server -XX:+UseG1GC 
-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 
-XX:+DisableExplicitGC -Djava.awt.headless=true 
-Xloggc:/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails 
-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
-Dcom.sun.management.jmxremote=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=10.60.68.122 
-Dcom.sun.management.jmxremote.rmi.port=7203 -Djava.net.preferIPv4Stack=true 
-Dcom.sun.management.jmxremote.port=7203 -Dkafka.logs.dir=/kafka/bin/../logs 
-Dlog4j.configuration=file:/kafka/bin/../config/log4j.properties -cp 
:/kafka/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/kafka/bin/../libs/argparse4j-0.5.0.jar:/kafka/bin/../libs/connect-api-0.10.1.0.jar:/kafka/bin/../libs/connect-file-0.10.1.0.jar:/kafka/bin/../libs/connect-json-0.10.1.0.jar:/kafka/bin/../libs/connect-runtime-0.10.1.0.jar:/kafka/bin/../libs/guava-18.0.jar:/kafka/bin/../libs/hk2-api-2.4.0-b34.jar:/kafka/bin/../libs/hk2-locator-2.4.0-b34.jar:/kafka/bin/../libs/hk2-utils-2.4.0-b34.jar:/kafka/bin/../libs/jackson-annotations-2.6.0.jar:/kafka/bin/../libs/jackson-core-2.6.3.jar:/kafka/bin/../libs/jackson-databind-2.6.3.jar:/kafka/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/kafka/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/kafka/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/kafka/bin/../libs/javassist-3.18.2-GA.jar:/kafka/bin/../libs/javax.annotation-api-1.2.jar:/kafka/bin/../libs/javax.inject-1.jar:/kafka/bin/../libs/javax.inject-2.4.0-b34.jar:/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/kafka/bin/../libs/jersey-client-2.22.2.jar:/kafka/bin/../libs/jersey-common-2.22.2.jar:/kafka/bin/../libs/jersey-container-servlet-2.22.2.jar:/kafka/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/kafka/bin/../libs/jersey-guava-2.22.2.jar:/kafka/bin/../libs/jersey-media-jaxb-2.22.2.jar:/kafka/bin/../libs/jersey-server-2.22.2.jar:/kafka/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-http-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-io-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-security-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-server-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/kafka/bin/../libs/jetty-util-9.2.15.v20160210.jar:/kafka/bin/../libs/jopt-simple-4.9.jar:/kafka/bin/../libs/kafka-clients-0.10.1.0.jar:/kafka/bin/../libs/kafka-log4j-appender-0.10.1.0.jar:/kafka/bin/../libs/kafka-streams-0.10.1.0.jar:/kafka/bin/../libs/kafka-streams-examples-0.10.1.0.jar:/kafka/bin/../libs/kafka-tools-0.10.1.0.jar:/kafka/bin/../libs/kafka_2.11-0.10.1.0-sources.jar:/kafka/bin/../libs/kafka_2.11-0.10.1.0-test-sources.jar:/kafka/bin/../libs/kafka_2.11-0.10.1.0.jar:/kafka/bin/../libs/log4j-1.2.17.jar:/kafka/bin/../libs/lz4-1.3.0.jar:/kafka/bin/../libs/metrics-core-2.2.0.jar:/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/kafka/bin/../libs/reflections-0.9.10.jar:/kafka/bin/../libs/rocksdbjni-4.9.0.jar:/kafka/bin/../libs/scala-library-2.11.8.jar:/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/kafka/bin/../libs/slf4j-api-1.7.21.jar:/kafka/bin/../libs/slf4j-log4j12-1.7.21.jar:/kafka/bin/../libs/snappy-java-1.1.2.6.jar:/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/kafka/bin/../libs/zkclient-0.9.jar:/kafka/bin/../libs/zookeeper-3.4.8.jar
 kafka.Kafka /kafka/config/server.properties

This is running inside a docker container.
Reporter: Xavier Lange


My cluster is unable to communicate to one of my brokers (Broker 1 in this 
case) and is spinning on logs:

{code}
[2016-11-29 19:05:08,659] WARN [ReplicaFetcherThread-0-1], Error in fetch 
kafka.server.ReplicaFetcherThread$FetchRequest@27aeb5f4 
(kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 10.60.68.122:9092 (id: 1 rack: null) failed
at 
kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
at 
kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706252#comment-15706252
 ] 

Onur Karaman commented on KAFKA-4447:
-

Hey [~guozhang]. I've been working on a detailed writeup for the controller 
redesign plan. So far it's got details on the protocols the controller 
participates in, current high-level controller design, all of the relevant 
components within the controller, problems with the current controller, and 
proposed improvements.

It's still a work-in-progress. Some of the proposed improvements are pretty 
detailed while others still need some work.

> Controller resigned but it also acts as a controller for a long time 
> -
>
> Key: KAFKA-4447
> URL: https://issues.apache.org/jira/browse/KAFKA-4447
> Project: Kafka
>  Issue Type: Improvement
>  Components: controller
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
> Environment: Linux Os
>Reporter: Json Tu
> Attachments: log.tar.gz
>
>
> We have a cluster with 10 nodes，and we execute following operation as below.
> 1.we execute some topic partition reassign from one node to other 9 nodes in 
> the cluster, and which triggered controller.
> 2.controller invoke PartitionsReassignedListener's handleDataChange and read 
> all partition reassign rules from the zk path, and executed all 
> onPartitionReassignment for all partition that match conditions.
> 3.but the controller is expired from zk, after what some nodes of 9 nodes 
> also expired from zk.
> 5.then controller invoke onControllerResignation to resigned as the 
> controller.
> we found after the controller is resigned, it acts as controller for about 3 
> minutes, which can be found in my attachment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4463) Setup travis-ci integration for ducktape tests

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created KAFKA-4463:
--

 Summary: Setup travis-ci integration for ducktape tests
 Key: KAFKA-4463
 URL: https://issues.apache.org/jira/browse/KAFKA-4463
 Project: Kafka
  Issue Type: Bug
  Components: system tests
Affects Versions: 0.10.0.1
Reporter: Raghav Kumar Gautam
Assignee: Raghav Kumar Gautam
 Fix For: 0.10.1.1






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4427) Skip topicGroups with no tasks

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-4427.
--
Resolution: Fixed

Issue resolved by pull request 2171
[https://github.com/apache/kafka/pull/2171]

> Skip topicGroups with no tasks
> --
>
> Key: KAFKA-4427
> URL: https://issues.apache.org/jira/browse/KAFKA-4427
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.2.0
>
>
> Currently the StreamPartitionAssignor's "assign" method does not handle cases 
> where we don't have tasks for a particular topic group. E.g., code like this 
> might give an NPE:
> "for (TaskId task : tasksByTopicGroup.get(topicGroupId))" 
> We need to handle the above cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Bill Bejeck]

+1 (non-binding)

Thanks,

Bill

On Tue, Nov 29, 2016 at 1:34 PM, Matthias J. Sax 
wrote:

> I’d like to start the voting process for KIP-93:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 93%3A+Improve+invalid+timestamp+handling+in+Kafka+Streams
>
> -Matthias
>
>

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Soumyajit Sahu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706222#comment-15706222
 ] 

Soumyajit Sahu commented on KAFKA-1194:
---

Thanks for confirming that [~haraldk]. Glad to be of help. I hope we can merge 
this into the trunk.
[~abhit011] Your issue seems to be your environment problem, and not related to 
the actual problem here. I will try to reach you over email later.

> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-1194.patch, Untitled.jpg, kafka-1194-v1.patch, 
> kafka-1194-v2.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 1516723
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>  at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
>  at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
>  at scala.collection.immutable.List.foreach(List.scala:76)
>  at kafka.log.Log.deleteOldSegments(Log.scala:418)
>  at 
> kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>  at 
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
>  at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>  at 
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
>  at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>  at kafka.log.LogManager.cleanupLogs(LogManager.scala:314)
>  at 
> kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143)
>  at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
>  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:724)
> I think this error happens because kafka tries to rename the log file when it 
> is still opened.  So we should close the file first before rename.
> The index file uses a special data structure, the MappedByteBuffer. Javadoc 
> describes it as:
> A mapped byte buffer and the file mapping that it represents remain valid 
> until the buffer itself is garbage-collected.
> Fortunately, I find a forceUnmap function in kafka code, and perhaps it can 
> be used to free the MappedByteBuffer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

Sorry, that link should be:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy



On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> Hi all,
> 
> I've been thinking about a KIP to improve the Kafka client's
> compatibility policy.  If you're interested, please check out: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
> 
> cheers,
> Colin

[DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

Hi all,

I've been thinking about a KIP to improve the Kafka client's
compatibility policy.  If you're interested, please check out: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy

cheers,
Colin

[jira] [Commented] (KAFKA-1696) Kafka should be able to generate Hadoop delegation tokens

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706194#comment-15706194
 ] 

Sriharsha Chintalapani commented on KAFKA-1696:
---

[~singhashish] [~omkreddy]  is working on it. I think its best to break this 
down into multiple JIRAs and distribute the work.

> Kafka should be able to generate Hadoop delegation tokens
> -
>
> Key: KAFKA-1696
> URL: https://issues.apache.org/jira/browse/KAFKA-1696
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Parth Brahmbhatt
>
> For access from MapReduce/etc jobs run on behalf of a user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4448) Add missing licenses to ducktape related files

[Raghav Kumar Gautam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghav Kumar Gautam resolved KAFKA-4448.

Resolution: Invalid

The problematic patch has been reverted, so the issue is no longer relevant.

> Add missing licenses to ducktape related files
> --
>
> Key: KAFKA-4448
> URL: https://issues.apache.org/jira/browse/KAFKA-4448
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
>
> Some files are missing licences, this is causing rat check to fail.
> {code}
> :rat
> Unknown license: 
> /Users/ewencp/kafka.git/gradle/wrapper/gradle-wrapper.properties
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/client1/__init__.py
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/core1/__init__.py
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/mirror_maker/__init__.py
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/replication/__init__.py
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/security1/__init__.py
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/security2/__init__.py
> Unknown license: 
> /Users/ewencp/kafka.git/tests/kafkatest/tests/upgrade/__init__.py
> Unknown license: /Users/ewencp/kafka.git/tests/travis/ssh/id_rsa
> Unknown license: /Users/ewencp/kafka.git/tests/travis/ssh/id_rsa.pub
> {code}
> This is a follow up of comments related to
> https://github.com/apache/kafka/pull/2064
> Thanks [~ewencp] for reporting the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4462) Improved Kafka Client Compatibility Policy

[Colin P. McCabe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe updated KAFKA-4462:
---
Description: A proposal to improve the compatibility policy of the Kafka 
client by supporting the combination of new client, old broker.  See 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
 for more details.

> Improved Kafka Client Compatibility Policy
> --
>
> Key: KAFKA-4462
> URL: https://issues.apache.org/jira/browse/KAFKA-4462
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.1.1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>
> A proposal to improve the compatibility policy of the Kafka client by 
> supporting the combination of new client, old broker.  See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
>  for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706166#comment-15706166
 ] 

Guozhang Wang commented on KAFKA-4405:
--

Thanks [~enothereska].

> Kafka consumer improperly send prefetch request
> ---
>
> Key: KAFKA-4405
> URL: https://issues.apache.org/jira/browse/KAFKA-4405
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: ysysberserk
>
> Now kafka consumer has added max.poll.records to limit the count of messages 
> return by poll().
> According to KIP-41, to implement  max.poll.records, the prefetch request 
> should only be sent when the total number of retained records is less than 
> max.poll.records.
> But in the code of 0.10.0.1 , the consumer will send a prefetch request if it 
> retained any records and never check if total number of retained records is 
> less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message 
> fetched , the poll() loop will send a lot of requests than expected and will 
> have more and more records fetched and stored in memory before they can be 
> consumed.
> So before sending a  prefetch request , the consumer must check if total 
> number of retained records is less than max.poll.records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #1061

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-4462) Improved Kafka Client Compatibility Policy

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-4462:
--

 Summary: Improved Kafka Client Compatibility Policy
 Key: KAFKA-4462
 URL: https://issues.apache.org/jira/browse/KAFKA-4462
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 0.10.1.1
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-4271:
---
Comment: was deleted

(was: [~yuyan] Thanks for the pointer, but when I look at the script 
{{kafka-server-start.bat}} the heap option is already set to 512M 
([here|https://github.com/apache/kafka/blob/trunk/bin/windows/zookeeper-server-start.bat]).
 And it's failing for me with this setting.)

> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.
> Update: This issue seems to occur on 32-bit Windows only. I tried this on a 
> Windows 32-bit VM. On a 64-bit machine I did not get the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706138#comment-15706138
 ] 

Vahid Hashemian commented on KAFKA-4271:


[~yuyan] Thanks for the pointer, but when I look at the script 
{{kafka-server-start.bat}} the heap option is already set to 512M 
([here|https://github.com/apache/kafka/blob/trunk/bin/windows/zookeeper-server-start.bat]).
 And it's failing for me with this setting.

> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.
> Update: This issue seems to occur on 32-bit Windows only. I tried this on a 
> Windows 32-bit VM. On a 64-bit machine I did not get the error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)