Re: [VOTE] 1.1.1 RC3

[Brett Rann]

+1 (non binding)
rolling upgrade of shared staging multitenacy (200+ consumer groups)
cluster from 1.1.0 to 1.1.1-rc3 using the kafka_2.11-1.1.1.tgz artifact.
cluster looks healthy after upgrade. Lack of burrow lag suggests consumers
are still happy, and incoming messages remains the same.

On Mon, Jul 9, 2018 at 8:36 AM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  > - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  > - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  > - Process txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  > - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  > -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  > - Fix
> NullPointerException when querying global state store
> KAFKA-6879  > - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  > - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  > - Log cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  > - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  > - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  > -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  > -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> 
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> 
>
> *** Please download, test and vote by Thursday, July 12, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
> 
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> 
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
> 
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
> 
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc3
> 
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
> 
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
> 
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/162
> 
>  >*
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
> 
>
> Please test and verify the release 

Re: Kafka disk recommendation for production cluster

[M. Manna]

Dong Lin's KIP -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-112%3A+Handle+disk+failure+for+JBOD

Should give you some ideas.

On 11 July 2018 at 14:31, Ali Nazemian  wrote:

> Hi All,
>
> I was wondering what the disk recommendation is for Kafka cluster? Is it
> acceptable to use RAID0 in the case that replication is 3? We are running
> on a cloud infrastructure and disk failure is addressed at another level,
> so the chance of single disk failure would be very low. Besides, our
> version of Kafka does not fully support JBOD as even a single disk failure
> can stop a Kafka broker, so we will not get the actual benefit of using
> JBOD anyway. However, I am not quite sure how software raid acts in this
> situation as there is no option to use HW Raid on cloud.
>
> Regards,
> Ali
>

Re: [VOTE] 1.1.1 RC3

[Jakub Scholz]

+1 (non-binbding) ... built from source, run tests and used it with several
of my applications without any problems.

Thanks & Regards
Jakub


On Mon, Jul 9, 2018 at 12:36 AM Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937  - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  - Process
> txn
> completion asynchronously to avoid deadlock
> KAFKA-6893  - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  - Fix
> NullPointerException when querying global state store
> KAFKA-6879  - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  - Log
> cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747  - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, July 12, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc3/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc3 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc3
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/162
> *
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/1.1/156/
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
>
> Regards,
> Dong
>

Re: [VOTE] 2.0.0 RC2

[Jakub Scholz]

+1 (non-binbding) ... I built the RC2 from source, run tests and used it
with several of my applications without any problems.

Thanks & Regards
Jakub

On Tue, Jul 10, 2018 at 7:17 PM Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the third candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new  KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
> A few notable highlights:
>
>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
>(KIP-277)
>- SASL/OAUTHBEARER implementation (KIP-255)
>- Improved quota communication and customization of quotas (KIP-219,
>KIP-257)
>- Efficient memory usage for down conversion (KIP-283)
>- Fix log divergence between leader and follower during fast leader
>failover (KIP-279)
>- Drop support for Java 7 and remove deprecated code including old scala
>clients
>- Connect REST extension plugin, support for externalizing secrets and
>improved error handling (KIP-285, KIP-297, KIP-298 etc.)
>- Scala API for Kafka Streams and other Streams API improvements
>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
>
> Release notes for the 2.0.0 release:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
>
>
> *** Please download, test and vote by Friday, July 13, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
>
>
> * Maven artifacts to be voted upon:
>
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
>
> http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> https://github.com/apache/kafka/tree/2.0.0-rc2
>
>
>
> * Documentation:
>
> http://kafka.apache.org/20/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/20/protocol.html
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/72/
>
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
>
>
> /**
>
>
> Thanks,
>
>
> Rajini
>

Re: [VOTE] 2.0.0 RC2

[Vahid S Hashemian]

+1 (non-binding)

Built executables from source and ran quickstart (Ubuntu / Java 8)

Thanks!
--Vahid




From:   Brett Rann 
To: d...@kafka.apache.org
Cc: Users , kafka-clients 

Date:   07/10/2018 09:53 PM
Subject:Re: [VOTE] 2.0.0 RC2



+1 (non binding)
rolling upgrade of tiny shared staging multitenacy (200+ consumer groups)
cluster from 1.1 to 2.0.0-rc1 to 2.0.0-rc2. cluster looks healthy after
upgrade. Lack of burrow lag suggests consumers are still happy, and
incoming messages remains the same.  Will monitor.

On Wed, Jul 11, 2018 at 3:17 AM Rajini Sivaram 
wrote:

> Hello Kafka users, developers and client-developers,
>
>
> This is the third candidate for release of Apache Kafka 2.0.0.
>
>
> This is a major version release of Apache Kafka. It includes 40 new KIPs
> and
>
> several critical bug fixes. Please see the 2.0.0 release plan for more
> details:
>
> 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820

> <
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>
>
>
> A few notable highlights:
>
> - Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for CreateTopics
> (KIP-277)
> - SASL/OAUTHBEARER implementation (KIP-255)
> - Improved quota communication and customization of quotas (KIP-219,
> KIP-257)
> - Efficient memory usage for down conversion (KIP-283)
> - Fix log divergence between leader and follower during fast leader
> failover (KIP-279)
> - Drop support for Java 7 and remove deprecated code including old scala
> clients
> - Connect REST extension plugin, support for externalizing secrets and
> improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> - Scala API for Kafka Streams and other Streams API improvements
> (KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>
>
> Release notes for the 2.0.0 release:
>
> 
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html

> <
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/RELEASE_NOTES.html
>
>
>
> *** Please download, test and vote by Friday, July 13, 4pm PT
>
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
>
> 
http://kafka.apache.org/KEYS

> <
http://kafka.apache.org/KEYS
>
>
>
> * Release artifacts to be voted upon (source and binary):
>
> 
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/

> <
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/
>
>
>
> * Maven artifacts to be voted upon:
>
> 
https://repository.apache.org/content/groups/staging/

> <
https://repository.apache.org/content/groups/staging/
>
>
>
> * Javadoc:
>
> 
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/

> <
http://home.apache.org/~rsivaram/kafka-2.0.0-rc2/javadoc/
>
>
>
> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>
> 
https://github.com/apache/kafka/tree/2.0.0-rc2

> <
https://github.com/apache/kafka/tree/2.0.0-rc2
>
>
>
>
> * Documentation:
>
> 
http://kafka.apache.org/20/documentation.html

> <
http://kafka.apache.org/20/documentation.html
>
>
>
> * Protocol:
>
> 
http://kafka.apache.org/20/protocol.html

> <
http://kafka.apache.org/20/protocol.html
>
>
>
> * Successful Jenkins builds for the 2.0 branch:
>
> Unit/integration tests: 
https://builds.apache.org/job/kafka-2.0-jdk8/72/

> <
https://builds.apache.org/job/kafka-2.0-jdk8/72/
>
>
> System tests:
> 
https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/

> <
https://jenkins.confluent.io/job/system-test-kafka/job/2.0/27/
>
>
>
> /**
>
>
> Thanks,
>
>
> Rajini
>


-- 

Brett Rann

Senior DevOps Engineer


Zendesk International Ltd

395 Collins Street, Melbourne VIC 3000 Australia

Mobile: +61 (0) 418 826 017

Re: Apache Kafka QuickStart

[Vahid S Hashemian]

Hi Nicholas,

The quickstart is meant to run in terminals. The two commands in Step 2 
should be run in different terminals unless you're sending the ZooKeeper 
process to the backgroud.
If you are facing particular errors please share so we can better assist 
you.

Thanks.
--Vahid




From:   Nicholas Chang 
To: "users@kafka.apache.org" 
Date:   07/11/2018 05:33 AM
Subject:Apache Kafka QuickStart



Hi,
I am new to Apache Kafka and I am trying to work on the QuickStart but run 
into problem in Step 2. After executing the first command to start 
zookeeper, do i have to open a Terminal to run the Kafka Server? I even 
try How To Install Apache Kafka on Ubuntu 14.04 | DigitalOcean also cannot 
get pass step 6. I am using Ubuntu 16.04 LTS. I look forward to receiving 
your reply soon.


| 
| 
| 
|  |  |

 |

 |
| 
|  | 
How To Install Apache Kafka on Ubuntu 14.04 | DigitalOcean

Apache Kafka is a popular distributed message broker designed to handle 
large volumes of real-time data efficien...
 |

 |

 |


Regards,Nicholas Chang

Kafka Streams processor node metrics process rate with multiple stream threads

[Sam Lendle]

Hello!

Using kafka-streams 1.1.0, I noticed when I sum the process rate metric for a 
given processor node, the rate is many times higher than the number of incoming 
messages. Digging further, it looks like the rate metric associated with each 
thread in a given application instance is always the same, and if I average by 
instance and then sum the rates, I recover the incoming message rate.  So it 
looks like the rate metric for each stream thread is actually the reporting the 
rate for all threads on the instance.

Is this a known issue, or am I misusing the metric? I’m not sure if this 
affects other metrics, but it does look like the average latency metric is 
identical for all threads on the same instance, so I suspect it does.

Thanks,
Sam

Partition assignment doesn't look right

[c . stanoiu]

Hello,
I have a kafka cluster (version 1.0.1) with two brokers.
I have four topics on this cluster  (w, x, y, z) with replication factor 2 and 
2 partitions each.
To this cluster I connect with two consumers using the kafka-streams api 
version 1.0.1.

Like so:

 @Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
public StreamsConfig kStreamsConfigs() {
Map props = new HashMap<>();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrapServers);
props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
props.put(StreamsConfig.TIMESTAMP_EXTRACTOR_CLASS_CONFIG, 
WallclockTimestampExtractor.class.getName());

props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, 
LogAndContinueExceptionHandler.class);
return new StreamsConfig(props);
}

@Bean(name = "WTable")
public KTablewKTable(ApplicationContext 
applicationContext, StreamsBuilder kStreamBuilder, WStore wStore) throws 
ClassNotFoundException {
KTable table = kStreamBuilder.table(wTopic, 
Consumed.with(Serdes.String(), MySerdes.wSerde()), 
Materialized.with(Serdes.String(), 
MySerdes.wSerde()).as(wStore.getStoreName()));
table.toStream().foreach((key, value) -> 
applicationContext.getBean(SomeService.class).processWEvent(key, value));
return table;
}

   @Bean(name = "XTable")
   ...

   @Bean(name = "YTable")
   ...

   @Bean(name = "ZTable")
public KTablezKTable(ApplicationContext 
applicationContext, StreamsBuilder kStreamBuilder, ZStore zStore) throws 
ClassNotFoundException {
KTable table = kStreamBuilder.table(zTopic, 
Consumed.with(Serdes.String(), MySerdes.zSerde()), 
Materialized.with(Serdes.String(), 
MySerdes.zSerde()).as(zStore.getStoreName()));
table.toStream().foreach((key, value) -> 
applicationContext.getBean(SomeService.class).processZEvent(key, value));
return table;
}

I would expect the two consumer applications I start would connect to all 4 
topics and each one will consume from one partition of those topics like so:
Consumer1 would maybe connect wTopic-0, xTopic-1, yTopic-0, zTopic-0
Consumer2 would maybe connect wTopic-1, xTopic-0, yTopic-1, zTopic-1

but what actually happens is:
Consumer1 connects wTopic-0, xTopic-1, xTopic-0, zTopic-0 (consumer 1 is not 
even consuming from yTopic)
Consumer2 connects wTopic-1, yTopic-0, yTopic-1, zTopic-1 (consumer 2 is not 
even consuming from xTopic)

Is there a way to address this problem?

PS: I have only noticed this problem when I connect to more than three topics

Re: Http streaming using Kafka+Akka

[Boris Lublinsky]

Yes, reactive Kafka

Boris Lublinsky
FDP Architect
boris.lublin...@lightbend.com
https://www.lightbend.com/

> On Jul 11, 2018, at 9:46 AM, Pulkit Manchanda  wrote:
> 
> Thanks Boris.
> Are you using Alpakka for kafka -Akka integration?
> 
> On Wed, Jul 11, 2018 at 9:56 AM, Boris Lublinsky <
> boris.lublin...@lightbend.com> wrote:
> 
>> This works fine, we (Lightbend) are using this approach all over the place
>> 
>> Boris Lublinsky
>> FDP Architect
>> boris.lublin...@lightbend.com
>> https://www.lightbend.com/
>> 
>>> On Jul 11, 2018, at 8:53 AM, Pulkit Manchanda 
>> wrote:
>>> 
>>> Hi All,
>>> 
>>> I want to build a datapipeline with the following design. Can please
>> anyone
>>> advice me that is it feasible to do? Or there are better options.
>>> 
>>> 
>>> HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) -->
>> (kafka
>>> Stream Producer)(using Kafka Streaming) --> (Kafka Stream
>> Consumer)(using Spark
>>> structured streaming)
>>> 
>>> 
>>> DPContext --> (Kafka Stream Producer)(using Spark structured
>>> streaming) --> (Kafka
>>> Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using
>> AKKA
>>> HTTP Streaming) --> HTTP Streams
>>> 
>>> Thanks
>>> Pulkit
>> 
>> 

Re: cleanup.policy - doesn't accept compact,delete

[Guozhang Wang]

You do not need the brackets, try keep the string value as "compact,delete".


Guozhang

On Tue, Jul 10, 2018 at 8:22 PM, Jayaraman, AshokKumar (CCI-Atlanta-CON) <
ashokkumar.jayara...@cox.com> wrote:

> Hi Matthias,
>
> Kept as [compact,delete] and still got the same exception.
>
> Thanks & Regards,
>
> Ashok
>
> -Original Message-
> From: Matthias J. Sax [mailto:matth...@confluent.io]
> Sent: Tuesday, July 10, 2018 10:09 PM
> To: users@kafka.apache.org
> Subject: Re: cleanup.policy - doesn't accept compact,delete
>
> Try to remove the space after the comma.
>
> -Matthias
>
> On 7/10/18 10:43 AM, Jayaraman, AshokKumar (CCI-Atlanta-CON) wrote:
> > Hi,
> >
> > When we try to use the same (square brackets), the internal topics are
> failing to get created.  Any suggestions?
> >
> > changelogConfig.put("cleanup.policy", "[compact, delete]");
> >
> >
> > org.apache.kafka.streams.errors.StreamsException: Could not create
> topic stream_digital_01-hrly-changelog.
> > at org.apache.kafka.streams.processor.internals.
> InternalTopicManager.makeReady(InternalTopicManager.java:137)
> > at org.apache.kafka.streams.processor.internals.
> StreamPartitionAssignor.prepareTopic(StreamPartitionAssignor.java:655)
> > at org.apache.kafka.streams.processor.internals.
> StreamPartitionAssignor.assign(StreamPartitionAssignor.java:463)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:358)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:520)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator.access$1100(AbstractCoordinator.java:93)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.
> java:472)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.
> java:455)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator$CoordinatorResponseHandler.
> onSuccess(AbstractCoordinator.java:822)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator$CoordinatorResponseHandler.
> onSuccess(AbstractCoordinator.java:802)
> > at org.apache.kafka.clients.consumer.internals.
> RequestFuture$1.onSuccess(RequestFuture.java:204)
> > at org.apache.kafka.clients.consumer.internals.
> RequestFuture.fireSuccess(RequestFuture.java:167)
> > at org.apache.kafka.clients.consumer.internals.
> RequestFuture.complete(RequestFuture.java:127)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(
> ConsumerNetworkClient.java:563)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.firePendingCompletedRequests(
> ConsumerNetworkClient.java:390)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.poll(ConsumerNetworkClient.java:293)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364)
> > at org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
> > at org.apache.kafka.clients.consumer.internals.
> ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
> > at org.apache.kafka.clients.consumer.KafkaConsumer.
> pollOnce(KafkaConsumer.java:1149)
> > at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> KafkaConsumer.java:1115)
> > at org.apache.kafka.streams.processor.internals.
> StreamThread.pollRequests(StreamThread.java:827)
> > at org.apache.kafka.streams.processor.internals.
> StreamThread.runOnce(StreamThread.java:784)
> > at org.apache.kafka.streams.processor.internals.
> StreamThread.runLoop(StreamThread.java:750)
> > at
> > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamTh
> > read.java:720) Caused by: java.util.concurrent.ExecutionException:
> > org.apache.kafka.common.errors.UnknownServerException: Invalid value
> [compact for configuration cleanup.policy: String must be one of: compact,
> delete
> > at org.apache.kafka.common.internals.KafkaFutureImpl.
> wrapAndThrow(KafkaFutureImpl.java:45)
> > at org.apache.kafka.common.internals.KafkaFutureImpl.
> access$000(KafkaFutureImpl.java:32)
> > at org.apache.kafka.common.internals.KafkaFutureImpl$
> SingleWaiter.await(KafkaFutureImpl.java:89)
> > at org.apache.kafka.common.internals.KafkaFutureImpl.get(
> KafkaFutureImpl.java:258)
> > 

Re: Questions about state stores and KSQL

[Guozhang Wang]

Hello Jonathan,

At the very high-level, KSQL statements is compiled into a Kafka Streams
topology for execution. And the concept "state stores" are for Kafka
Streams, not for KSQL, where inside the topology for those processor nodes
that need stateful processing, like Joins, one or more state stores would
be associated with the nodes.

Back to your example, this KSQL statement will be compiled into a Kafka
Streams that roughly looks like this:

--

Kafka topic that defines stream "topics" --> source node --> join node
(queries the "users-state" store, as generated below) --> sink node --> Kafka
topic that defines stream "orders_enriched"

Kafka topic that defines table "users" --> source node --> materialization
node (associated with a state store, let's name it "users-state")

--

That is, one state store will be used to materialize the table changelog
stream for "users", where the other stream's record will query on.

In Kafka Streams, you can query a state store following the interactive
query mechanism:

https://kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html

It is not supported in KSQL yet.



Guozhang




On Wed, Jul 11, 2018 at 1:41 AM, Jonathan Roy <
jonathan@caldera.com.invalid> wrote:

> Hi Kafka users,
>
> I am very new to Kafka and more globally to stream processing, and am
> trying to understand some of the concepts used by Kafka. From what I
> understand, a key-value state store is created on each processor node that
> performs stateful operations such as aggregations or joins. Let’s take an
> example. I have an ‘orders’ stream and a ‘users’ table, and I want to
> enrich the orders events with the corresponding users information, using
> the KSQL CLI:
>
> CREATE STREAM orders_enriched AS SELECT o.id , o.article,
> o.quantity, o.userId, u.name, u.address, u.email FROM orders o LEFT JOIN
> users u ON o.userId = u.id ;
>
> Where is located the state store in this case? What will it contain
> exactly? Is it possible to query it from another node?
>
> Thanks beforehand for your help!
>
> Jonathan




-- 
-- Guozhang

Re: Http streaming using Kafka+Akka

[Pulkit Manchanda]

Thanks Boris.
Are you using Alpakka for kafka -Akka integration?

On Wed, Jul 11, 2018 at 9:56 AM, Boris Lublinsky <
boris.lublin...@lightbend.com> wrote:

> This works fine, we (Lightbend) are using this approach all over the place
>
> Boris Lublinsky
> FDP Architect
> boris.lublin...@lightbend.com
> https://www.lightbend.com/
>
> > On Jul 11, 2018, at 8:53 AM, Pulkit Manchanda 
> wrote:
> >
> > Hi All,
> >
> > I want to build a datapipeline with the following design. Can please
> anyone
> > advice me that is it feasible to do? Or there are better options.
> >
> >
> > HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) -->
> (kafka
> > Stream Producer)(using Kafka Streaming) --> (Kafka Stream
> Consumer)(using Spark
> > structured streaming)
> >
> >
> > DPContext --> (Kafka Stream Producer)(using Spark structured
> > streaming) --> (Kafka
> > Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using
> AKKA
> > HTTP Streaming) --> HTTP Streams
> >
> > Thanks
> > Pulkit
>
>

Kafka for Vehicle location tracking

[Yash Ganthe]

Hi,

I am new to Kafka and hence would like to validate the following design.
Imagine, a vehicle is being tracked by multiple people. V1 is tracked by
U1, U2, U3. When V1 moves U1, U2, U3 should be
notified and updated. U1, U2, U3 would be tracking several other vehicles
too.

Let me know if there is any issue with this design:

   - Vehicle communicates its location to a port where a program listens
   and logs to a Topic.
   - Vehicle location is treated as a stream.
   - The processor generates another stream that has User as the key. For
   each Vehicle, multiple records are logged to the user stream.
   - Another processor consumes the User stream and pushes the new vehicle
   locations subscribed by that user over a websocket.

Is that the recommended approach?

Regards,
Yash

Re: Http streaming using Kafka+Akka

[Boris Lublinsky]

This works fine, we (Lightbend) are using this approach all over the place

Boris Lublinsky
FDP Architect
boris.lublin...@lightbend.com
https://www.lightbend.com/

> On Jul 11, 2018, at 8:53 AM, Pulkit Manchanda  wrote:
> 
> Hi All,
> 
> I want to build a datapipeline with the following design. Can please anyone
> advice me that is it feasible to do? Or there are better options.
> 
> 
> HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) --> (kafka
> Stream Producer)(using Kafka Streaming) --> (Kafka Stream Consumer)(using 
> Spark
> structured streaming)
> 
> 
> DPContext --> (Kafka Stream Producer)(using Spark structured
> streaming) --> (Kafka
> Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using AKKA
> HTTP Streaming) --> HTTP Streams
> 
> Thanks
> Pulkit

Http streaming using Kafka+Akka

[Pulkit Manchanda]

Hi All,

I want to build a datapipeline with the following design. Can please anyone
advice me that is it feasible to do? Or there are better options.


HTTP Streams --> (HTTP stream consumer)(using AKKA HTTP Streaming) --> (kafka
Stream Producer)(using Kafka Streaming) --> (Kafka Stream Consumer)(using Spark
structured streaming)


DPContext --> (Kafka Stream Producer)(using Spark structured
streaming) --> (Kafka
Stream Consumer)(using AKKA Streaming) -->(HTTP stream Producer)(using AKKA
HTTP Streaming) --> HTTP Streams

Thanks
Pulkit

Kafka disk recommendation for production cluster

[Ali Nazemian]

Hi All,

I was wondering what the disk recommendation is for Kafka cluster? Is it
acceptable to use RAID0 in the case that replication is 3? We are running
on a cloud infrastructure and disk failure is addressed at another level,
so the chance of single disk failure would be very low. Besides, our
version of Kafka does not fully support JBOD as even a single disk failure
can stop a Kafka broker, so we will not get the actual benefit of using
JBOD anyway. However, I am not quite sure how software raid acts in this
situation as there is no option to use HW Raid on cloud.

Regards,
Ali

Apache Kafka QuickStart

[Nicholas Chang]

Hi,
I am new to Apache Kafka and I am trying to work on the QuickStart but run into 
problem in Step 2. After executing the first command to start zookeeper, do i 
have to open a Terminal to run the Kafka Server? I even try How To Install 
Apache Kafka on Ubuntu 14.04 | DigitalOcean also cannot get pass step 6. I am 
using Ubuntu 16.04 LTS. I look forward to receiving your reply soon.


| 
| 
| 
|  |  |

 |

 |
| 
|  | 
How To Install Apache Kafka on Ubuntu 14.04 | DigitalOcean

Apache Kafka is a popular distributed message broker designed to handle large 
volumes of real-time data efficien...
 |

 |

 |


Regards,Nicholas Chang

Questions about state stores and KSQL

[Jonathan Roy]

Hi Kafka users,

I am very new to Kafka and more globally to stream processing, and am trying to 
understand some of the concepts used by Kafka. From what I understand, a 
key-value state store is created on each processor node that performs stateful 
operations such as aggregations or joins. Let’s take an example. I have an 
‘orders’ stream and a ‘users’ table, and I want to enrich the orders events 
with the corresponding users information, using the KSQL CLI:

CREATE STREAM orders_enriched AS SELECT o.id , o.article, 
o.quantity, o.userId, u.name, u.address, u.email FROM orders o LEFT JOIN users 
u ON o.userId = u.id ;

Where is located the state store in this case? What will it contain exactly? Is 
it possible to query it from another node?

Thanks beforehand for your help!

Jonathan 

Re: Steadily increasing open file descriptors on one node a cluster

[Michael Rauter]

Furher investigations:

I have compared open files/connections of the different nodes. Same count in 
real open files (data dir files) and established connections on all nodes.

But the affected node has a lot of "CLOSE_WAIT" connections (many thousends) to 
IPs of external clients (no specific ip). The other nodes less than 10.


Hi,

I’m running a Kafka cluster with many topics and constant input of data.
The cluster is running for over one year but now (since 2 weeks) there is 
one node where I see a steady increase of open file descriptors of the Kafka 
server process.
All other nodes have a constant number of this metric. Topics/partitions 
are distributed equal over all nodes, same hardware.

The open file limit was reached last week and the node worked normally 
after restart and recovery…but since the restart the file descriptors are 
increasing again..

Any idea or things to do to find out more?

Version: 0.10.2.1

Thanks,
Michael

Re: cleanup.policy - doesn't accept compact,delete

[Manikumar]

we need to use square brackets only for command line tool.
In your code, you just need to supply  "compact,delete" string to props
object.

On Wed, Jul 11, 2018 at 8:52 AM Jayaraman, AshokKumar (CCI-Atlanta-CON) <
ashokkumar.jayara...@cox.com> wrote:

> Hi Matthias,
>
> Kept as [compact,delete] and still got the same exception.
>
> Thanks & Regards,
>
> Ashok
>
> -Original Message-
> From: Matthias J. Sax [mailto:matth...@confluent.io]
> Sent: Tuesday, July 10, 2018 10:09 PM
> To: users@kafka.apache.org
> Subject: Re: cleanup.policy - doesn't accept compact,delete
>
> Try to remove the space after the comma.
>
> -Matthias
>
> On 7/10/18 10:43 AM, Jayaraman, AshokKumar (CCI-Atlanta-CON) wrote:
> > Hi,
> >
> > When we try to use the same (square brackets), the internal topics are
> failing to get created.  Any suggestions?
> >
> > changelogConfig.put("cleanup.policy", "[compact, delete]");
> >
> >
> > org.apache.kafka.streams.errors.StreamsException: Could not create topic
> stream_digital_01-hrly-changelog.
> > at
> org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:137)
> > at
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.prepareTopic(StreamPartitionAssignor.java:655)
> > at
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:463)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:358)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:520)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:93)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:472)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:455)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:822)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:802)
> > at
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
> > at
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
> > at
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:563)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:390)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:293)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364)
> > at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
> > at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149)
> > at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
> > at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:827)
> > at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:784)
> > at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
> > at
> > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamTh
> > read.java:720) Caused by: java.util.concurrent.ExecutionException:
> > org.apache.kafka.common.errors.UnknownServerException: Invalid value
> [compact for configuration cleanup.policy: String must be one of: compact,
> delete
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> > at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:258)
> >