Build failed in Jenkins: kafka-trunk-jdk10 #314

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H23 (ubuntu xenial) in workspace

[jira] [Created] (KAFKA-7203) Improve Streams StickyTaskAssingor

Guozhang Wang created KAFKA-7203: Summary: Improve Streams StickyTaskAssingor Key: KAFKA-7203 URL: https://issues.apache.org/jira/browse/KAFKA-7203 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-7144) Kafka Streams doesn't properly balance partition assignment

[ https://issues.apache.org/jira/browse/KAFKA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-7144. -- Resolution: Fixed Fix Version/s: 2.1.0 > Kafka Streams doesn't properly balance

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

Current voting result is shown below. +2 (binding) 1) Jason Gustafson 2) Gwen Shapira +4 (non-binding) 1) Colin McCabe 2) Manikumar 3) Dong Lin 4) Vahid S Hashemian Need one more binding vote please Cheers, Chia-Ping On 2018/07/16 08:10:51, Chia-Ping Tsai wrote: > hi folks, > > The

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

I have updated the KIP . Changes to the KIP: - Removed topics() from the Public Interface and Proposed Changes sections. - Added topics() to the

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

Hey James, Ted, @James - Thanks for showing me some of the changes, that was informative. * *Log Cleaner Thread Revival* - I also acknowledge that could be useful. My concern is that if the thread has died, there is most likely something wrong with either the disk or the software and since both

Re: Processor API StateStore and Recovery with State Machines question.

Hello Adam, I figured that rather than answering your questions one-by-one, I'd give you a more general explanation between consumer offset commits, changelog and state store. If you have a state store update processor, the state store maintenance workflow is this: 1) updating the state store:

Re: [VOTE] 2.0.0 RC3

+1 (non-binding) Built from source and ran quickstart successfully with both Java 8 and Java 9 on Ubuntu. Thanks Rajini! --Vahid From: Rajini Sivaram To: dev , Users , kafka-clients Date: 07/24/2018 08:33 AM Subject:[VOTE] 2.0.0 RC3 Hello Kafka users, developers and

Re: [DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

Hi Ron, Thanks for the suggestions. I have applied them to the KIP. On Tue, Jul 24, 2018 at 1:39 PM Ron Dagostino wrote: > Hi Stanislav. The statement "New config option for OAuthBearerLoginModule" > is technically incorrect; it should be "New config option for default, > unsecured bearer

[jira] [Created] (KAFKA-7202) Support multiple auto-generated docs formats

Joel Hamill created KAFKA-7202: -- Summary: Support multiple auto-generated docs formats Key: KAFKA-7202 URL: https://issues.apache.org/jira/browse/KAFKA-7202 Project: Kafka Issue Type: New

Re: [DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

Hi Stanislav. The statement "New config option for OAuthBearerLoginModule" is technically incorrect; it should be "New config option for default, unsecured bearer tokens" since that is what provides the functionality (as opposed to the login module, which does not). Also, please state that

NotCoordinatorForGroupException in a long loop

Hi, org.apache.kafka.common.errors.NotCoordinatorForGroupException: This is not the correct coordinator for this group. When there is an under-replication partition, it causes our stream-application to stall with NotCoorindatorForGroupException. This retriableException goes on in a loop for a

dev and prod cluster consumption rate are different

Hi, I have a kafka stream application that processes 250K events per minute in our dev kafka cluster but in production using the same binary, the performance degrades dramatically to a few thousands per minute. What could possibly be wrong? 1. server.properties in dev and prod, kafka brokers

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi Jason, Thanks for clarifying. So if we are going to continue supporting the empty group id as before (with only an addition of a deprecation warning), and disable enable.auto.commit for the new default (null) group id on the client side, do we really need to bump up the OffsetCommit

[jira] [Resolved] (KAFKA-7195) StreamStreamJoinIntegrationTest fails in 2.0 Jenkins

[ https://issues.apache.org/jira/browse/KAFKA-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-7195. -- Resolution: Fixed Fix Version/s: 2.0.0 > StreamStreamJoinIntegrationTest fails in 2.0

[jira] [Resolved] (KAFKA-2318) replica manager repeatedly tries to fetch from partitions already moved during controlled shutdown

[ https://issues.apache.org/jira/browse/KAFKA-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2318. -- Resolution: Auto Closed Closing inactive issue. Please reopen if the issue still exists in newer

[jira] [Resolved] (KAFKA-3040) Broker didn't report new data after change in leader

[ https://issues.apache.org/jira/browse/KAFKA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3040. -- Resolution: Auto Closed Closing inactive issue. Please reopen if the issue still exists in newer

[jira] [Resolved] (KAFKA-2890) Strange behaviour during partitions reassignment.

[ https://issues.apache.org/jira/browse/KAFKA-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2890. -- Resolution: Auto Closed Closing inactive issue. Please reopen if the issue still exists in newer

[jira] [Created] (KAFKA-7201) Optimize repartition operations

Bill Bejeck created KAFKA-7201: -- Summary: Optimize repartition operations Key: KAFKA-7201 URL: https://issues.apache.org/jira/browse/KAFKA-7201 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-2968) Refactor ReplicaManager to be optimal for o.a.k.c requests

[ https://issues.apache.org/jira/browse/KAFKA-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2968. -- Resolution: Fixed Closing this as scala request/responses classes are removed and ReplicaManager is 

[jira] [Created] (KAFKA-7200) Preserve serdes used

Bill Bejeck created KAFKA-7200: -- Summary: Preserve serdes used Key: KAFKA-7200 URL: https://issues.apache.org/jira/browse/KAFKA-7200 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-2969) Refactor MetadataCache to be optimal for o.a.k.c requests

[ https://issues.apache.org/jira/browse/KAFKA-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2969. -- Resolution: Fixed Closing this as scala request/responses classes are removed and MetadataCache

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

Thanks, Dong! In terms of the background retention thread reading the last entry of timeindex files, there are two scenarios we need to consider: 1. Most segments in each log have passed their TTL. In this case, the background thread has to read the last entry of the timeindex files for

[jira] [Resolved] (KAFKA-2075) Validate that all kafka.api requests has been removed and clean up compatibility code

[ https://issues.apache.org/jira/browse/KAFKA-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2075. -- Resolution: Fixed Old scala request/reponse classes are removed as part of KAFKA-2983: > Validate

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Lucas, I agree, if we want to go forward with a separate controller plane and data plane and completely isolate them, having a separate port for controller with a separate Acceptor and a Processor sounds ideal to me. Thanks, Mayuresh On Mon, Jul 23, 2018 at 11:04 PM Becket Qin wrote: > Hi

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

We should not remove it immediately in the up coming 2.1 release. Usually we first mark an API as deprecated, and consider removing it only after it has been deprecated for at least one major release period. Guozhang On Mon, Jul 23, 2018 at 7:40 PM, Nishanth Pradeep wrote: > Sounds good to me

[jira] [Created] (KAFKA-7199) Support BigInteger data type

Jiri Pechanec created KAFKA-7199: Summary: Support BigInteger data type Key: KAFKA-7199 URL: https://issues.apache.org/jira/browse/KAFKA-7199 Project: Kafka Issue Type: Improvement

[VOTE] 2.0.0 RC3

Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 2.0.0. This is a major version release of Apache Kafka. It includes 40 new KIPs and several critical bug fixes. Please see the 2.0.0 release plan for more details:

Re: [DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

Hey everybody, I have updated the KIP to reflect the latest changes as best as I could. If there aren't more suggestions, I intent to start the [VOTE] thread tomorrow. Best, Stanislav On Tue, Jul 24, 2018 at 6:34 AM Ron Dagostino wrote: > Hi Stanislav. Could you update the KIP to reflect the

[jira] [Created] (KAFKA-7198) Enhance KafkaStreams start method javadoc

Bill Bejeck created KAFKA-7198: -- Summary: Enhance KafkaStreams start method javadoc Key: KAFKA-7198 URL: https://issues.apache.org/jira/browse/KAFKA-7198 Project: Kafka Issue Type: Improvement

Re: Didn't receive video stream data into spark through kafka

Yes, I have double check that. On 24 July 2018 at 19:20, Aman Rastogi wrote: > Is your topic same in both the case? > > On Tue, 24 Jul 2018, 19:15 Biswajit Ghosh, > wrote: > > > Hi team, > > > > I got an issue while integrating with the spark streaming using pyspark, > I > > did receive the

Re: Didn't receive video stream data into spark through kafka

Is your topic same in both the case? On Tue, 24 Jul 2018, 19:15 Biswajit Ghosh, wrote: > Hi team, > > I got an issue while integrating with the spark streaming using pyspark, I > did receive the video stream data in a different consumer subscribe to the > same topic. > > Works fine with this

Didn't receive video stream data into spark through kafka

Hi team, I got an issue while integrating with the spark streaming using pyspark, I did receive the video stream data in a different consumer subscribe to the same topic. Works fine with this command : *./kafka-console-consumer.sh --bootstrap-server :9092 --topic spark-streaming-consumer

Re: [DISCUSS] KIP-342 Add Customizable SASL extensions to OAuthBearer authentication

Hi Stanislav. Could you update the KIP to reflect the latest definition of SaslExtensions and confirm or correct the impact it has to the SCRAM-related classes? I'm not sure if the currently-described impact is still accurate. Also, could you mention the changes to

[jira] [Created] (KAFKA-7197) Release a milestone build for Scala 2.13.0 M3

Martynas Mickevičius created KAFKA-7197: --- Summary: Release a milestone build for Scala 2.13.0 M3 Key: KAFKA-7197 URL: https://issues.apache.org/jira/browse/KAFKA-7197 Project: Kafka

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

As James pointed out in his reply, topic-partition name can be long. It is not necessary to repeat the topic name for each of its partitions. How about the following format: topic-name1-{partition1, partition2, etc} That is, topic name only appears once. Cheers On Mon, Jul 23, 2018 at 9:08 PM

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

Let me kick this off with a few starting points that I would like to generate some discussion on. 1) It seems to me that I will need to repartition the data twice - once on the foreign key, and once back to the primary key. Is there anything I am missing here? 2) I believe I will also need to

Build failed in Jenkins: kafka-trunk-jdk8 #2837

See Changes: [ismael] MINOR: Remove unused TopicAndPartition usage in tests (#5419) [rajinisivaram] KAFKA-7194; Fix buffer underflow if onJoinComplete is retried after

Jenkins build is back to normal : kafka-2.0-jdk8 #90

See

Jenkins build is back to normal : kafka-trunk-jdk10 #311

See

[jira] [Resolved] (KAFKA-7194) Error deserializing assignment after rebalance

[ https://issues.apache.org/jira/browse/KAFKA-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-7194. --- Resolution: Fixed Reviewer: Konstantine Karantasis Fix Version/s: 2.0.0 >

[jira] [Resolved] (KAFKA-2963) Replace server internal usage of TopicAndPartition with TopicPartition

[ https://issues.apache.org/jira/browse/KAFKA-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2963. -- Resolution: Fixed This has been fixed in various cleanups > Replace server internal usage of

Build failed in Jenkins: kafka-trunk-jdk8 #2836

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H32 (ubuntu xenial) in workspace

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

Hi Stanislav! Thanks for this KIP! I agree that it would be good if the LogCleaner were more tolerant of errors. Currently, as you said, once it dies, it stays dead. Things are better now than they used to be. We have the metric

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Lucas, Yes, I agree that a dedicated end to end control flow would be ideal. Thanks, Jiangjie (Becket) Qin On Tue, Jul 24, 2018 at 1:05 PM, Lucas Wang wrote: > Thanks for the comment, Becket. > So far, we've been trying to avoid making any request handler thread > special. > But if we