[jira] [Resolved] (KAFKA-7369) Retry when possible in AdminClient.listConsumerGroups
[
https://issues.apache.org/jira/browse/KAFKA-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-7369.
Resolution: Fixed
Fix Version/s: 2.1.0
2.0.1
> Retry when possible in AdminClient.listConsumerGroups
> -
>
> Key: KAFKA-7369
> URL: https://issues.apache.org/jira/browse/KAFKA-7369
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.0.1, 2.1.0
>
>
> Currently we do not retry ListGroups requests when they fail due to retriable
> errors. For example, this is causing some instability in
> `kafka.admin.ListConsumerGroupTest.testListConsumerGroups`.
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorLoadInProgressException: Error
> listing groups on localhost:43001 (id: 0 rack: null)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.listGroups(ConsumerGroupCommand.scala:132)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [jira] [Created] (KAFKA-7370) Enhance FileConfigProvider to read a directory
Do you want to raise a KIP and explain the motivation behind it ? Please ignore if you’re doing that already. Thanks , On Sat, 1 Sep 2018 at 00:15, Robert Yokota (JIRA) wrote: > Robert Yokota created KAFKA-7370: > > > Summary: Enhance FileConfigProvider to read a directory > Key: KAFKA-7370 > URL: https://issues.apache.org/jira/browse/KAFKA-7370 > Project: Kafka > Issue Type: Improvement > Components: config > Affects Versions: 2.0.0 > Reporter: Robert Yokota > Assignee: Robert Yokota > > > Currently FileConfigProvider can read a Properties file as a set of > key-value pairs. This enhancement is to augment FileConfigProvider so that > it can also read a directory, where the file names are the keys and the > corresponding file contents are the values. > > This will allow for easier integration with secret management systems > where each secret is often an individual file, such as in Docker and > Kubernetes. > > > > -- > This message was sent by Atlassian JIRA > (v7.6.3#76005) >
[jira] [Created] (KAFKA-7370) Enhance FileConfigProvider to read a directory
Robert Yokota created KAFKA-7370: Summary: Enhance FileConfigProvider to read a directory Key: KAFKA-7370 URL: https://issues.apache.org/jira/browse/KAFKA-7370 Project: Kafka Issue Type: Improvement Components: config Affects Versions: 2.0.0 Reporter: Robert Yokota Assignee: Robert Yokota Currently FileConfigProvider can read a Properties file as a set of key-value pairs. This enhancement is to augment FileConfigProvider so that it can also read a directory, where the file names are the keys and the corresponding file contents are the values. This will allow for easier integration with secret management systems where each secret is often an individual file, such as in Docker and Kubernetes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7369) Retry when possible in AdminClient.listConsumerGroups
Jason Gustafson created KAFKA-7369:
--
Summary: Retry when possible in AdminClient.listConsumerGroups
Key: KAFKA-7369
URL: https://issues.apache.org/jira/browse/KAFKA-7369
Project: Kafka
Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Currently we do not retry ListGroups requests when they fail due to retriable
errors. For example, this is causing some instability in
`kafka.admin.ListConsumerGroupTest.testListConsumerGroups`.
{code}
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.CoordinatorLoadInProgressException: Error
listing groups on localhost:43001 (id: 0 rack: null)
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.listGroups(ConsumerGroupCommand.scala:132)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7287) Set open ACL permissions for old consumer znode path
[ https://issues.apache.org/jira/browse/KAFKA-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-7287. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 1.1.2 Also merged the PR to 1.1 branch. > Set open ACL permissions for old consumer znode path > > > Key: KAFKA-7287 > URL: https://issues.apache.org/jira/browse/KAFKA-7287 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Manikumar >Assignee: Manikumar >Priority: Major > Fix For: 1.1.2, 2.0.1, 2.1.0 > > > Old consumer znode path should have open ACL permissions in kerberized > environment. This got missed in kafkaZkClient changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7368) Support joining Windowed KTables
John Roesler created KAFKA-7368: --- Summary: Support joining Windowed KTables Key: KAFKA-7368 URL: https://issues.apache.org/jira/browse/KAFKA-7368 Project: Kafka Issue Type: Improvement Components: streams Reporter: John Roesler Currently, there is no good way to join two `KTable, V>`, aka windowed KTables. They are KTable, so they have a `join` operator available, but it currently will use a regular KeyValue store instead of a Windowed store, so it will grow without bound and new windows enter. One option is to convert both KTables into KStream, and join them (which is a windowed join), and then convert them back into KTables for further processing, but this is an awkward way to accomplish an apparently straightforward task. It should instead be possible to directly support it, but the trick will be to make it impossible to accidentally use a window store for normal (aka non-windowed) KTables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7367) Verify that Streams never creates RocksDB stores unless they are needed
John Roesler created KAFKA-7367: --- Summary: Verify that Streams never creates RocksDB stores unless they are needed Key: KAFKA-7367 URL: https://issues.apache.org/jira/browse/KAFKA-7367 Project: Kafka Issue Type: Improvement Components: streams Reporter: John Roesler We have gotten some reports of Streams creating RocksDB stores unnecessarily for stateless processes. We can and should verify this doesn't happen by creating integration tests for *every* stateless operator that verify that after processing, the state directory is still empty. These tests could potentially be backported as far as we care to so that we can identify and fix potential unnecessary stores in past versions as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-4988) JVM crash when running on Alpine Linux
[
https://issues.apache.org/jira/browse/KAFKA-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler resolved KAFKA-4988.
-
Resolution: Won't Fix
I think this issue is out of our hands.
If you think there is something for us to do, please feel free to reopen the
ticket and comment.
Thanks,
-John
> JVM crash when running on Alpine Linux
> --
>
> Key: KAFKA-4988
> URL: https://issues.apache.org/jira/browse/KAFKA-4988
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Vincent Rischmann
>Priority: Minor
>
> I'm developing my Kafka Streams application using Docker and I run my jars
> using the official openjdk:8-jre-alpine image.
> I'm just starting to use windowing and now the JVM crashes because of an
> issue with RocksDB I think.
> It's trivial to fix on my part, just use the debian jessie based image.
> However, it would be cool if alpine was supported too since its docker images
> are quite a bit less heavy
> {quote}
> Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError:
> /tmp/librocksdbjni3285995384052305662.so: Error loading shared library
> ld-linux-x86-64.so.2: No such file or directory (needed by
> /tmp/librocksdbjni3285995384052305662.so)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
> at java.lang.Runtime.load0(Runtime.java:809)
> at java.lang.System.load(System.java:1086)
> at
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
> at
> org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
> at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
> at org.rocksdb.RocksDB.(RocksDB.java:35)
> at org.rocksdb.Options.(Options.java:22)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:115)
> at
> org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:148)
> at
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:39)
> at
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore$7.run(MeteredKeyValueStore.java:100)
> at
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
> at
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:131)
> at
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:62)
> at
> org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:86)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:141)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f60f34ce088, pid=1, tid=0x7f60f3705ab0
> #
> # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64
> compressed oops)
> #
[jira] [Resolved] (KAFKA-6033) Kafka Streams does not work with musl-libc (alpine linux)
[ https://issues.apache.org/jira/browse/KAFKA-6033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-6033. - Resolution: Won't Fix This is unfortunately out of our hands. If you think I'm wrong about this, please reopen the ticket. Thanks, -John > Kafka Streams does not work with musl-libc (alpine linux) > - > > Key: KAFKA-6033 > URL: https://issues.apache.org/jira/browse/KAFKA-6033 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 0.11.0.0 > Environment: Alpine 3.6 >Reporter: Jeffrey Zampieron >Priority: Major > > Using the released version of kafka fails on alpine based images b/c of > rocksdb using the jni and failing to load. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] KIP-347: Enable batching in FindCoordinatorRequest
@Guozhang Wang Could you review this again when you have time? Thanks! -Yishun On Wed, Aug 29, 2018 at 11:57 AM Yishun Guan wrote: > > Hi, because I have made some significant changes on this design, so I > want to reopen the discussion on this KIP: > https://cwiki.apache.org/confluence/x/CgZPBQ > > Thanks, > Yishun > On Thu, Aug 16, 2018 at 5:06 PM Yishun Guan wrote: > > > > I see! Thanks! > > > > On Thu, Aug 16, 2018, 4:35 PM Guozhang Wang wrote: > >> > >> It is not implemented, but should not be hard to do so (and again you do > >> NOT have to do that in this KIP, I'm bringing this up so that you can help > >> thinking about the process). > >> > >> Quoting from Colin's comment: > >> > >> " > >> The pattern is that you would try to send a request for more than one > >> group, and then you would get an UnsupportedVersionException (nothing would > >> be sent on the wire, this is purely internal to the code). > >> Then your code would handle the UVE by creating requests with an older > >> version that only had one group each. > >> " > >> > >> > >> Guozhang > >> > >> > >> On Wed, Aug 15, 2018 at 4:44 PM, Yishun Guan wrote: > >> > >> > Hi, I am looking into AdminClient.scala and AdminClient.java, and also > >> > looking into ApiVersionRequest.java and ApiVersionResponse.java, but I > >> > don't see anywhere contains to logic of the one-to-one mapping from > >> > version > >> > to version, am i looking at the right place? > >> > > >> > On Mon, Aug 13, 2018 at 1:30 PM Guozhang Wang wrote: > >> > > >> > > Regarding 3): Today we do not have this logic with the existing client, > >> > > because defer the decision about the version to use (we always assume > >> > that > >> > > an new versioned request need to be down-converted to a single old > >> > > versioned request: i.e. an one-to-one mapping), but in principle, we > >> > should > >> > > be able to modify the client make it work. > >> > > > >> > > Again this is not necessarily need to be included in this KIP, but I'd > >> > > recommend you to look into AdminClient implementations around the > >> > > ApiVersionRequest / Response and think about how that logic can be > >> > modified > >> > > in the follow-up PR of this KIP. > >> > > > >> > > > >> > > > >> > > Guozhang > >> > > > >> > > On Mon, Aug 13, 2018 at 12:55 PM, Yishun Guan > >> > > wrote: > >> > > > >> > > > @Guozhang, thank you so much! > >> > > > 1. I agree, fixed. > >> > > > 2. Added. > >> > > > 3. I see, that is something that I haven't think about. How does > >> > > > Kafka > >> > > > handle other api's different version problem now? So we have a > >> > > > specific > >> > > > convertor that convect a new version request to a old version one for > >> > > each > >> > > > API (is this what the ApiVersionsRequest supposed to do, does it only > >> > > > handle the detection or it should handle the conversion too)? What > >> > > > will > >> > > be > >> > > > the consequence of not having such a transformer but the version is > >> > > > incompatible? > >> > > > > >> > > > Best, > >> > > > Yishun > >> > > > > >> > > > On Sat, Aug 11, 2018 at 11:27 AM Guozhang Wang > >> > > wrote: > >> > > > > >> > > > > Hello Yishun, > >> > > > > > >> > > > > Thanks for the proposed KIP. I made a pass over the wiki and here > >> > > > > are > >> > > > some > >> > > > > comments: > >> > > > > > >> > > > > 1. "DESCRIBE_GROUPS_RESPONSE_MEMBER_V0", why we need to encode the > >> > full > >> > > > > schema for the "COORDINATOR_GROUPIDS_KEY_NAME" field? Note it > >> > includes > >> > > a > >> > > > > lot of fields such as member id that is not needed for this case. I > >> > > > think a > >> > > > > "new ArrayOf(String)" for the group ids should be sufficient. > >> > > > > > >> > > > > 2. "schemaVersions" of the "FindCoordinatorRequest" needs to > >> > > > > include > >> > > > > FIND_COORDINATOR_REQUEST_V3 as well. > >> > > > > > >> > > > > 3. One thing you may need to consider is that, in the adminClient > >> > > > > to > >> > > > handle > >> > > > > broker compatibility, how to transform a new (v3) request to a > >> > > > > bunch > >> > of > >> > > > > (v2) requests if it detects the broker is still in old version and > >> > > hence > >> > > > > cannot support v3 request (this logic is already implemented via > >> > > > > ApiVersionsRequest in AdminClient, but may need to be extended to > >> > > handle > >> > > > > one-to-many mapping of different versions). > >> > > > > > >> > > > > This is not sth. that you need to implement under this KIP, but I'd > >> > > > > recommend you think about this earlier than later and see if it may > >> > > > affect > >> > > > > this proposal. > >> > > > > > >> > > > > > >> > > > > Guozhang > >> > > > > > >> > > > > > >> > > > > On Sat, Aug 11, 2018 at 10:54 AM, Yishun Guan > >> > > wrote: > >> > > > > > >> > > > > > Hi, thank you Ted! I have addressed your comments: > >> > > > > > > >> > > > > > 1. Added more descriptions about later optimization. > >> > > > > > 2. Yes, I will
Re: [VOTE] KIP-363: Make FunctionConversions private
Yeah, let's go ahead and do that to minimize confusion and to stick to the formal process. Sorry for the run-around. Thanks, -John On Fri, Aug 31, 2018 at 11:27 AM Joan Goyeau wrote: > Ah ok I didn't know we need multiple binding vote. > Should I send again a new email with the updated KIP-366 title? > > Thanks > > On Wed, 29 Aug 2018 at 21:14 John Roesler wrote: > > > Hey Joan, > > > > It looks like you've updated the KIP to "Accepted", but I only count one > > binding vote (Guozhang). Ted, Attila, Bill, and myself are all > non-binding > > votes. > > > > For reference, these are all folks who hold binding votes: > > https://kafka.apache.org/committers . Obviously, they don't all take > note > > of every KIP, so we sometimes have to keep pinging the thread with > > reminders that we are waiting on binding votes. > > > > Also, people muddied the water by responding "+1" to this thread, but > it's > > customary to start a new thread entitled "[VOTE] KIP-366: Make > > FunctionConversions private" to let people know when the voting has > > actually started. > > > > Thanks, > > -John > > > > On Mon, Aug 27, 2018 at 3:44 PM Joan Goyeau wrote: > > > > > John, no this is for internal use only. > > > I fact I expect this object to go away with the drop of Scala 2.11 > since > > in > > > Scala 2.12 we have support for SAM. > > > > > > Thanks > > > > > > On Mon, 27 Aug 2018 at 15:41 John Roesler wrote: > > > > > > > Hey Joan, > > > > > > > > I was thinking more about this... Do any of the conversions in > > > > FunctionConversions convert to types that are used in the public > Scala > > > > interface? > > > > > > > > If you've already checked, then carry on. > > > > > > > > Otherwise, we should leave public any that might be in use. > > > > > > > > Thanks, > > > > -John > > > > > > > > On Sat, Aug 25, 2018 at 12:19 PM Joan Goyeau > wrote: > > > > > > > > > Thanks Attila, it's done. > > > > > > > > > > On Sat, 25 Aug 2018 at 02:57 Ted Yu wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > On Fri, Aug 24, 2018 at 5:17 PM Attila Sasvári < > > asasv...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi there, > > > > > > > > > > > > > > There is a conflicting KIP with the same number, see > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Allow+performance+tools+to+print+final+results+to+output+file > > > > > > > > > > > > > > Its discussion was started earlier, on August 23 > > > > > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg91132.html > > > and > > > > > KIP > > > > > > > page already includes it: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > > > Please update KIP number to resolve the conflict. > > > > > > > > > > > > > > Apart from this, +1 (non-binding) and thanks for the KIP! > > > > > > > > > > > > > > Regards, > > > > > > > - Attila > > > > > > > > > > > > > > > > > > > > > Guozhang Wang (időpont: 2018. aug. 24., > P, > > > > 20:26) > > > > > > ezt > > > > > > > írta: > > > > > > > > > > > > > > > +1 from me (binding). > > > > > > > > > > > > > > > > On Fri, Aug 24, 2018 at 11:24 AM, Joan Goyeau < > j...@goyeau.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > As pointed out in this comment #5539 (comment) > > > > > > > > > < > > > https://github.com/apache/kafka/pull/5539#discussion_r212380648 > > > > > > > > > > > > "This > > > > > > > > > class was already defaulted to public visibility, and we > > can't > > > > > > retract > > > > > > > it > > > > > > > > > now, without a KIP.", the object FunctionConversions is > only > > of > > > > > > > internal > > > > > > > > > use and therefore should be private to the lib only so that > > we > > > > can > > > > > do > > > > > > > > > changes without going through KIP like this one. > > > > > > > > > > > > > > > > > > Please make your vote. > > > > > > > > > > > > > > > > > > On Fri, 24 Aug 2018 at 19:14 John Roesler < > j...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I'm also in favor of this. I don't think it's > controversial > > > > > either. > > > > > > > > > Should > > > > > > > > > > we just move to a vote? > > > > > > > > > > > > > > > > > > > > On Thu, Aug 23, 2018 at 7:01 PM Guozhang Wang < > > > > > wangg...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1. > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 23, 2018 at 12:47 PM, Ted Yu < > > > > yuzhih...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > In the Motivation section, you can quote the comment > > from > > > > > pull > > > > > > > > > request > > > > > > > > > > so > > > > > > > > > > > > that reader doesn't have to click through. > > > > > > > > > > > > >
Re: [VOTE] KIP-358: Migrate Streams API to Duration instead of long ms times
Hi Nikolay, You can start a PR any time, but we cannot per it (and probably won't do serious reviews) until after the KIP is voted and approved. Sometimes people start a PR during discussion just to help provide more context, but it's not required (and can also be distracting because the KIP discussion should avoid implementation details). Let's wait one more day for any other comments and plan to start the vote on Monday if there are no other debates. Once you start the vote, you have to leave it up for at least 72 hours, and it requires 3 binding votes to pass. Only Kafka Committers have binding votes (https://kafka.apache.org/committers). Thanks, -John On Fri, Aug 31, 2018 at 11:09 AM Bill Bejeck wrote: > Hi Nickolay, > > Thanks for the clarification. > > -Bill > > On Fri, Aug 31, 2018 at 11:59 AM Nikolay Izhikov > wrote: > > > Hello, John. > > > > This is my first KIP, so, please, help me with kafka development process. > > > > Should I start to work on PR now? Or should I wait for a "+1" from > > commiters? > > > > В Пт, 31/08/2018 в 10:33 -0500, John Roesler пишет: > > > I see. I guess that once we are in the PR-reviewing phase, we'll be in > a > > > better position to see what else can/should be done, and we can talk > > about > > > follow-on work at that time. > > > > > > Thanks for the clarification, > > > -John > > > > > > On Fri, Aug 31, 2018 at 1:19 AM Nikolay Izhikov > > wrote: > > > > > > > Hello, Bill > > > > > > > > > In the "Proposed Changes" section, there is "Try to reduce the > > > > > > > > visibility of methods in next tickets" does that mean eventual > > deprecation > > > > and removal? > > > > > > > > 1. Some methods will become deprecated. I think they will be removed > in > > > > the future. > > > > You can find list of deprecated methods in KIP. > > > > > > > > 2. Some internal methods can't be deprecated or hid from the user for > > now. > > > > I was trying to say that we should research possibility to reduce > > > > visibility of *internal* methods that are *public* now. > > > > That kind of changes is out of the scope of current KIP, so we have > to > > do > > > > it in the next tickets. > > > > > > > > I don't expect that internal methods will be removed. > > > > > > > > В Чт, 30/08/2018 в 18:59 -0400, Bill Bejeck пишет: > > > > > Sorry for chiming in late, there was a lot of detail to catch up > on. > > > > > > > > > > Overall I'm +1 in the KIP. But I do have one question about the > KIP > > in > > > > > regards to Matthias's comments about defining dual use. > > > > > > > > > > In the "Proposed Changes" section, there is "Try to reduce the > > visibility > > > > > of methods in next tickets" does that mean eventual deprecation and > > > > > > > > removal? > > > > > I thought we were aiming to keep the dual use methods? Or does that > > imply > > > > > we'll strive for more clear delineation between DSL and internal > use? > > > > > > > > > > Thanks, > > > > > Bill > > > > > > > > > > > > > > > > > > > > On Thu, Aug 30, 2018 at 5:59 PM Nikolay Izhikov < > nizhi...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > John, thank you. > > > > > > > > > > > > I've updated KIP. > > > > > > > > > > > > > > > > > > Dear commiters, please take a look and share your opinion. > > > > > > > > > > > > В Чт, 30/08/2018 в 14:58 -0500, John Roesler пишет: > > > > > > > Oh! I missed one minor thing: UnlimitedWindows doesn't need to > > set > > > > > > > > grace > > > > > > > (it currently does not either). > > > > > > > > > > > > > > Otherwise, it looks good to me! > > > > > > > > > > > > > > Thanks so much, > > > > > > > -John > > > > > > > > > > > > > > On Thu, Aug 30, 2018 at 5:30 AM Nikolay Izhikov < > > nizhi...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello, John. > > > > > > > > > > > > > > > > I've updated KIP according on your comments. > > > > > > > > Please, take a look. > > > > > > > > > > > > > > > > Are we ready to vot now? > > > > > > > > > > > > > > > > В Ср, 29/08/2018 в 14:51 -0500, John Roesler пишет: > > > > > > > > > Hey Nikolay, sorry for the silence. I'm taking another look > > at > > > > > > > > the > > > > > > > > > > > > KIP > > > > > > > > > before voting... > > > > > > > > > > > > > > > > > > > > > > > > > > >1. I think the Window constructor should actually be > > > > > > > > protected. I > > > > > > > > > > > > > > > > don't > > > > > > > > >know if we need a constructor that takes Instant, but if > > we > > > > > > > > do add > > > > > > > > > > > > > > > > one, it > > > > > > > > >should definitely be protected. > > > > > > > > >2. `long JoinWindows#size()` is overridden from `long > > > > > > > > > > > > Windows#size()`, > > > > > > > > >and should not be deprecated. Also, I don't think we > need > > a > > > > > > > > > > > > `Duration > > > > > > > > >JoinWindows#windowSize()`. > > > > > > > > >3. Likewise, `JoinWindows#windowsFor()` is overridden > from > > > > > > > > >
[jira] [Created] (KAFKA-7366) topic level segment.bytes and segment.ms not taking effect immediately
Jun Rao created KAFKA-7366: -- Summary: topic level segment.bytes and segment.ms not taking effect immediately Key: KAFKA-7366 URL: https://issues.apache.org/jira/browse/KAFKA-7366 Project: Kafka Issue Type: Bug Affects Versions: 1.1.0 Reporter: Jun Rao It used to be that topic level configs such as segment.bytes takes effect immediately. Because of KAFKA-6324 in 1.1, those configs now only take effect after the active segment has rolled. The relevant part of KAFKA-6324 is that in Log.maybeRoll, the checking of the segment rolling is moved to LogSegment.shouldRoll(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] KIP-363: Make FunctionConversions private
Ah ok I didn't know we need multiple binding vote. Should I send again a new email with the updated KIP-366 title? Thanks On Wed, 29 Aug 2018 at 21:14 John Roesler wrote: > Hey Joan, > > It looks like you've updated the KIP to "Accepted", but I only count one > binding vote (Guozhang). Ted, Attila, Bill, and myself are all non-binding > votes. > > For reference, these are all folks who hold binding votes: > https://kafka.apache.org/committers . Obviously, they don't all take note > of every KIP, so we sometimes have to keep pinging the thread with > reminders that we are waiting on binding votes. > > Also, people muddied the water by responding "+1" to this thread, but it's > customary to start a new thread entitled "[VOTE] KIP-366: Make > FunctionConversions private" to let people know when the voting has > actually started. > > Thanks, > -John > > On Mon, Aug 27, 2018 at 3:44 PM Joan Goyeau wrote: > > > John, no this is for internal use only. > > I fact I expect this object to go away with the drop of Scala 2.11 since > in > > Scala 2.12 we have support for SAM. > > > > Thanks > > > > On Mon, 27 Aug 2018 at 15:41 John Roesler wrote: > > > > > Hey Joan, > > > > > > I was thinking more about this... Do any of the conversions in > > > FunctionConversions convert to types that are used in the public Scala > > > interface? > > > > > > If you've already checked, then carry on. > > > > > > Otherwise, we should leave public any that might be in use. > > > > > > Thanks, > > > -John > > > > > > On Sat, Aug 25, 2018 at 12:19 PM Joan Goyeau wrote: > > > > > > > Thanks Attila, it's done. > > > > > > > > On Sat, 25 Aug 2018 at 02:57 Ted Yu wrote: > > > > > > > > > +1 > > > > > > > > > > On Fri, Aug 24, 2018 at 5:17 PM Attila Sasvári < > asasv...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi there, > > > > > > > > > > > > There is a conflicting KIP with the same number, see > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Allow+performance+tools+to+print+final+results+to+output+file > > > > > > > > > > > > Its discussion was started earlier, on August 23 > > > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg91132.html > > and > > > > KIP > > > > > > page already includes it: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > Please update KIP number to resolve the conflict. > > > > > > > > > > > > Apart from this, +1 (non-binding) and thanks for the KIP! > > > > > > > > > > > > Regards, > > > > > > - Attila > > > > > > > > > > > > > > > > > > Guozhang Wang (időpont: 2018. aug. 24., P, > > > 20:26) > > > > > ezt > > > > > > írta: > > > > > > > > > > > > > +1 from me (binding). > > > > > > > > > > > > > > On Fri, Aug 24, 2018 at 11:24 AM, Joan Goyeau > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > As pointed out in this comment #5539 (comment) > > > > > > > > < > > https://github.com/apache/kafka/pull/5539#discussion_r212380648 > > > > > > > > > > "This > > > > > > > > class was already defaulted to public visibility, and we > can't > > > > > retract > > > > > > it > > > > > > > > now, without a KIP.", the object FunctionConversions is only > of > > > > > > internal > > > > > > > > use and therefore should be private to the lib only so that > we > > > can > > > > do > > > > > > > > changes without going through KIP like this one. > > > > > > > > > > > > > > > > Please make your vote. > > > > > > > > > > > > > > > > On Fri, 24 Aug 2018 at 19:14 John Roesler > > > > > wrote: > > > > > > > > > > > > > > > > > I'm also in favor of this. I don't think it's controversial > > > > either. > > > > > > > > Should > > > > > > > > > we just move to a vote? > > > > > > > > > > > > > > > > > > On Thu, Aug 23, 2018 at 7:01 PM Guozhang Wang < > > > > wangg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1. > > > > > > > > > > > > > > > > > > > > On Thu, Aug 23, 2018 at 12:47 PM, Ted Yu < > > > yuzhih...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > In the Motivation section, you can quote the comment > from > > > > pull > > > > > > > > request > > > > > > > > > so > > > > > > > > > > > that reader doesn't have to click through. > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau < > > > > j...@goyeau.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > As pointed out in this comment #5539 (comment) > > > > > > > > > > > > < > > > > > > https://github.com/apache/kafka/pull/5539#discussion_r212380648 > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > object FunctionConversions is only of internal use > and > > > > > > therefore > > > >
Re: [VOTE] KIP-358: Migrate Streams API to Duration instead of long ms times
Hi Nickolay, Thanks for the clarification. -Bill On Fri, Aug 31, 2018 at 11:59 AM Nikolay Izhikov wrote: > Hello, John. > > This is my first KIP, so, please, help me with kafka development process. > > Should I start to work on PR now? Or should I wait for a "+1" from > commiters? > > В Пт, 31/08/2018 в 10:33 -0500, John Roesler пишет: > > I see. I guess that once we are in the PR-reviewing phase, we'll be in a > > better position to see what else can/should be done, and we can talk > about > > follow-on work at that time. > > > > Thanks for the clarification, > > -John > > > > On Fri, Aug 31, 2018 at 1:19 AM Nikolay Izhikov > wrote: > > > > > Hello, Bill > > > > > > > In the "Proposed Changes" section, there is "Try to reduce the > > > > > > visibility of methods in next tickets" does that mean eventual > deprecation > > > and removal? > > > > > > 1. Some methods will become deprecated. I think they will be removed in > > > the future. > > > You can find list of deprecated methods in KIP. > > > > > > 2. Some internal methods can't be deprecated or hid from the user for > now. > > > I was trying to say that we should research possibility to reduce > > > visibility of *internal* methods that are *public* now. > > > That kind of changes is out of the scope of current KIP, so we have to > do > > > it in the next tickets. > > > > > > I don't expect that internal methods will be removed. > > > > > > В Чт, 30/08/2018 в 18:59 -0400, Bill Bejeck пишет: > > > > Sorry for chiming in late, there was a lot of detail to catch up on. > > > > > > > > Overall I'm +1 in the KIP. But I do have one question about the KIP > in > > > > regards to Matthias's comments about defining dual use. > > > > > > > > In the "Proposed Changes" section, there is "Try to reduce the > visibility > > > > of methods in next tickets" does that mean eventual deprecation and > > > > > > removal? > > > > I thought we were aiming to keep the dual use methods? Or does that > imply > > > > we'll strive for more clear delineation between DSL and internal use? > > > > > > > > Thanks, > > > > Bill > > > > > > > > > > > > > > > > On Thu, Aug 30, 2018 at 5:59 PM Nikolay Izhikov > > > > > > > wrote: > > > > > > > > > John, thank you. > > > > > > > > > > I've updated KIP. > > > > > > > > > > > > > > > Dear commiters, please take a look and share your opinion. > > > > > > > > > > В Чт, 30/08/2018 в 14:58 -0500, John Roesler пишет: > > > > > > Oh! I missed one minor thing: UnlimitedWindows doesn't need to > set > > > > > > grace > > > > > > (it currently does not either). > > > > > > > > > > > > Otherwise, it looks good to me! > > > > > > > > > > > > Thanks so much, > > > > > > -John > > > > > > > > > > > > On Thu, Aug 30, 2018 at 5:30 AM Nikolay Izhikov < > nizhi...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello, John. > > > > > > > > > > > > > > I've updated KIP according on your comments. > > > > > > > Please, take a look. > > > > > > > > > > > > > > Are we ready to vot now? > > > > > > > > > > > > > > В Ср, 29/08/2018 в 14:51 -0500, John Roesler пишет: > > > > > > > > Hey Nikolay, sorry for the silence. I'm taking another look > at > > > > > > the > > > > > > > > > > KIP > > > > > > > > before voting... > > > > > > > > > > > > > > > > > > > > > > > >1. I think the Window constructor should actually be > > > > > > protected. I > > > > > > > > > > > > > > don't > > > > > > > >know if we need a constructor that takes Instant, but if > we > > > > > > do add > > > > > > > > > > > > > > one, it > > > > > > > >should definitely be protected. > > > > > > > >2. `long JoinWindows#size()` is overridden from `long > > > > > > > > > > Windows#size()`, > > > > > > > >and should not be deprecated. Also, I don't think we need > a > > > > > > > > > > `Duration > > > > > > > >JoinWindows#windowSize()`. > > > > > > > >3. Likewise, `JoinWindows#windowsFor()` is overridden from > > > > > > > >`Windows#windowsFor()` and should also not be deprecated, > and > > > > > > we > > > > > > > > > > also > > > > > > > > > > > > > > don't > > > > > > > >need a `Map windowsForTime(final Instant > > > > > > > > > > timestamp)` > > > > > > > >version. > > > > > > > >4. TimeWindowedDeserializer is a bit of a puzzle for me. > It > > > > > > > > > > actually > > > > > > > >looks like it's incorrectly implemented! I'm not sure if > we > > > > > > > > > > want/need > > > > > > > > > > > > > > to > > > > > > > >update any of its methods or constructors. > > > > > > > >5. TimeWindows: see my feedback on JoinWindows > > > > > > > >6. UnlimitedWindows: see my feedback on JoinWindows > > > > > > > >7. ReadOnlyWindowStore: the existing `long` methods > should be > > > > > > > >deprecated. (we should add `WindowStoreIterator fetch(K > > > > > > key, > > > > > > > > > > long > > > > > > > >timeFrom, long timeTo)` to WindowStore) > > > > > > > >8. SessionBytesStoreSupplier: Both of those methods
Re: [VOTE] KIP-358: Migrate Streams API to Duration instead of long ms times
Hello, John. This is my first KIP, so, please, help me with kafka development process. Should I start to work on PR now? Or should I wait for a "+1" from commiters? В Пт, 31/08/2018 в 10:33 -0500, John Roesler пишет: > I see. I guess that once we are in the PR-reviewing phase, we'll be in a > better position to see what else can/should be done, and we can talk about > follow-on work at that time. > > Thanks for the clarification, > -John > > On Fri, Aug 31, 2018 at 1:19 AM Nikolay Izhikov wrote: > > > Hello, Bill > > > > > In the "Proposed Changes" section, there is "Try to reduce the > > > > visibility of methods in next tickets" does that mean eventual deprecation > > and removal? > > > > 1. Some methods will become deprecated. I think they will be removed in > > the future. > > You can find list of deprecated methods in KIP. > > > > 2. Some internal methods can't be deprecated or hid from the user for now. > > I was trying to say that we should research possibility to reduce > > visibility of *internal* methods that are *public* now. > > That kind of changes is out of the scope of current KIP, so we have to do > > it in the next tickets. > > > > I don't expect that internal methods will be removed. > > > > В Чт, 30/08/2018 в 18:59 -0400, Bill Bejeck пишет: > > > Sorry for chiming in late, there was a lot of detail to catch up on. > > > > > > Overall I'm +1 in the KIP. But I do have one question about the KIP in > > > regards to Matthias's comments about defining dual use. > > > > > > In the "Proposed Changes" section, there is "Try to reduce the visibility > > > of methods in next tickets" does that mean eventual deprecation and > > > > removal? > > > I thought we were aiming to keep the dual use methods? Or does that imply > > > we'll strive for more clear delineation between DSL and internal use? > > > > > > Thanks, > > > Bill > > > > > > > > > > > > On Thu, Aug 30, 2018 at 5:59 PM Nikolay Izhikov > > > > wrote: > > > > > > > John, thank you. > > > > > > > > I've updated KIP. > > > > > > > > > > > > Dear commiters, please take a look and share your opinion. > > > > > > > > В Чт, 30/08/2018 в 14:58 -0500, John Roesler пишет: > > > > > Oh! I missed one minor thing: UnlimitedWindows doesn't need to set > > > > grace > > > > > (it currently does not either). > > > > > > > > > > Otherwise, it looks good to me! > > > > > > > > > > Thanks so much, > > > > > -John > > > > > > > > > > On Thu, Aug 30, 2018 at 5:30 AM Nikolay Izhikov > > > > > > > wrote: > > > > > > > > > > > Hello, John. > > > > > > > > > > > > I've updated KIP according on your comments. > > > > > > Please, take a look. > > > > > > > > > > > > Are we ready to vot now? > > > > > > > > > > > > В Ср, 29/08/2018 в 14:51 -0500, John Roesler пишет: > > > > > > > Hey Nikolay, sorry for the silence. I'm taking another look at > > > > the > > > > > > > > KIP > > > > > > > before voting... > > > > > > > > > > > > > > > > > > > > >1. I think the Window constructor should actually be > > > > protected. I > > > > > > > > > > > > don't > > > > > > >know if we need a constructor that takes Instant, but if we > > > > do add > > > > > > > > > > > > one, it > > > > > > >should definitely be protected. > > > > > > >2. `long JoinWindows#size()` is overridden from `long > > > > > > > > Windows#size()`, > > > > > > >and should not be deprecated. Also, I don't think we need a > > > > > > > > `Duration > > > > > > >JoinWindows#windowSize()`. > > > > > > >3. Likewise, `JoinWindows#windowsFor()` is overridden from > > > > > > >`Windows#windowsFor()` and should also not be deprecated, and > > > > we > > > > > > > > also > > > > > > > > > > > > don't > > > > > > >need a `Map windowsForTime(final Instant > > > > > > > > timestamp)` > > > > > > >version. > > > > > > >4. TimeWindowedDeserializer is a bit of a puzzle for me. It > > > > > > > > actually > > > > > > >looks like it's incorrectly implemented! I'm not sure if we > > > > > > > > want/need > > > > > > > > > > > > to > > > > > > >update any of its methods or constructors. > > > > > > >5. TimeWindows: see my feedback on JoinWindows > > > > > > >6. UnlimitedWindows: see my feedback on JoinWindows > > > > > > >7. ReadOnlyWindowStore: the existing `long` methods should be > > > > > > >deprecated. (we should add `WindowStoreIterator fetch(K > > > > key, > > > > > > > > long > > > > > > >timeFrom, long timeTo)` to WindowStore) > > > > > > >8. SessionBytesStoreSupplier: Both of those methods are > > > > "internal > > > > > > > > use > > > > > > >methods", so we should just leave them alone and not add new > > > > ones. > > > > > > >9. SessionStore: I don't think these are "external use" > > > > methods > > > > > > > > (only > > > > > > >ReadOnlySessionStore is used in IQ) maybe we should just leave > > > > > > > > them > > > > > > > > > > > > alone? > > > > > > >10.
Re: [jira] [Created] (KAFKA-7363) How num.stream.threads in streaming application influence memory consumption?
Hi Seweryn, It's a little hard to say. For one thing, extra threads have some overhead of their own, but I agree with you that the bulk of the extra memory would come from the extra throughput you're able to drive through the application. I haven't done any analysis of this before, so just reasoning about this (as opposed to speaking from experience): In the maximum case, doubling your thread count would double your memory usage. This is for an "ideal" CPU-bound process. In reality, there are shared resources, such as network and disk, that should prevent you from reaching this bound. In the minimum case, if the app is already saturating some other resource, like network, disk, or even memory, then increasing the thread count would not add an appreciable amount of memory. The reason is that if the app is saturating, say, the network already, then more threads doesn't change that fact, and you still can't increase the throughput. As far as a concrete answer to your question, I think you're unfortunately the only one with enough visibility to predict the memory load. It would be very dependent on your machines, network, the number of topics and partitions, the size of your records in each partition, what exactly your Streams app does, and even your broker configuration. However, I'd propose the following experimental strategy to try and get a handle on it: 1. start with one thread. Observe all the main resources (CPU, network i/o, disk i/o), but especially memory. For memory, pay particular attention to the memory used immediately after GC. You might want to turn on GC logging to help with this. 1b. observe these metrics for long enough for a stable trend to emerge. This might be hours or even a day. 2. add one more thread. Continue observing all the resources. As I said, in the ideal case, this should double your throughput and hence double your memory usage. Looking at how much all the extra metrics increase when you add the second thread should help you start building a model of the increase you should expect for each extra thread. 3. continue the experiment, adding one thread each time. At some point, you'll notice that the throughput/memory increase drops off when you add an extra thread. This means that you've saturated one or more other resource. The metrics for those resources should corroborate this. Note that, if nothing else, the CPU should become saturated once the number of threads is equal to the number of cores. Increasing the thread count much beyond this shouldn't help much. I hope this helps! On Fri, Aug 31, 2018 at 1:02 AM Seweryn Habdank-Wojewodzki (JIRA) < j...@apache.org> wrote: > Seweryn Habdank-Wojewodzki created KAFKA-7363: > - > > Summary: How num.stream.threads in streaming application > influence memory consumption? > Key: KAFKA-7363 > URL: https://issues.apache.org/jira/browse/KAFKA-7363 > Project: Kafka > Issue Type: Task > Reporter: Seweryn Habdank-Wojewodzki > > > Dears, > > How option _num.stream.threads_ in streaming application influence memory > consumption? > I see that by increasing num.stream.threads my application needs more > memory. > This is obvious, but it is not obvious how much I need to give it. Try and > error method does not work, as it seems to be highly dependen on forced > throughput. > I mean: higher load more memory is needed. > > Thanks for help and regards, > Seweryn. > > > > > -- > This message was sent by Atlassian JIRA > (v7.6.3#76005) >
Re: [VOTE] KIP-358: Migrate Streams API to Duration instead of long ms times
I see. I guess that once we are in the PR-reviewing phase, we'll be in a better position to see what else can/should be done, and we can talk about follow-on work at that time. Thanks for the clarification, -John On Fri, Aug 31, 2018 at 1:19 AM Nikolay Izhikov wrote: > Hello, Bill > > > In the "Proposed Changes" section, there is "Try to reduce the > visibility of methods in next tickets" does that mean eventual deprecation > and removal? > > 1. Some methods will become deprecated. I think they will be removed in > the future. > You can find list of deprecated methods in KIP. > > 2. Some internal methods can't be deprecated or hid from the user for now. > I was trying to say that we should research possibility to reduce > visibility of *internal* methods that are *public* now. > That kind of changes is out of the scope of current KIP, so we have to do > it in the next tickets. > > I don't expect that internal methods will be removed. > > В Чт, 30/08/2018 в 18:59 -0400, Bill Bejeck пишет: > > Sorry for chiming in late, there was a lot of detail to catch up on. > > > > Overall I'm +1 in the KIP. But I do have one question about the KIP in > > regards to Matthias's comments about defining dual use. > > > > In the "Proposed Changes" section, there is "Try to reduce the visibility > > of methods in next tickets" does that mean eventual deprecation and > removal? > > I thought we were aiming to keep the dual use methods? Or does that imply > > we'll strive for more clear delineation between DSL and internal use? > > > > Thanks, > > Bill > > > > > > > > On Thu, Aug 30, 2018 at 5:59 PM Nikolay Izhikov > wrote: > > > > > John, thank you. > > > > > > I've updated KIP. > > > > > > > > > Dear commiters, please take a look and share your opinion. > > > > > > В Чт, 30/08/2018 в 14:58 -0500, John Roesler пишет: > > > > Oh! I missed one minor thing: UnlimitedWindows doesn't need to set > grace > > > > (it currently does not either). > > > > > > > > Otherwise, it looks good to me! > > > > > > > > Thanks so much, > > > > -John > > > > > > > > On Thu, Aug 30, 2018 at 5:30 AM Nikolay Izhikov > > > > > > > wrote: > > > > > > > > > Hello, John. > > > > > > > > > > I've updated KIP according on your comments. > > > > > Please, take a look. > > > > > > > > > > Are we ready to vot now? > > > > > > > > > > В Ср, 29/08/2018 в 14:51 -0500, John Roesler пишет: > > > > > > Hey Nikolay, sorry for the silence. I'm taking another look at > the > > > > > > KIP > > > > > > before voting... > > > > > > > > > > > > > > > > > >1. I think the Window constructor should actually be > protected. I > > > > > > > > > > don't > > > > > >know if we need a constructor that takes Instant, but if we > do add > > > > > > > > > > one, it > > > > > >should definitely be protected. > > > > > >2. `long JoinWindows#size()` is overridden from `long > > > > > > Windows#size()`, > > > > > >and should not be deprecated. Also, I don't think we need a > > > > > > `Duration > > > > > >JoinWindows#windowSize()`. > > > > > >3. Likewise, `JoinWindows#windowsFor()` is overridden from > > > > > >`Windows#windowsFor()` and should also not be deprecated, and > we > > > > > > also > > > > > > > > > > don't > > > > > >need a `Map windowsForTime(final Instant > > > > > > timestamp)` > > > > > >version. > > > > > >4. TimeWindowedDeserializer is a bit of a puzzle for me. It > > > > > > actually > > > > > >looks like it's incorrectly implemented! I'm not sure if we > > > > > > want/need > > > > > > > > > > to > > > > > >update any of its methods or constructors. > > > > > >5. TimeWindows: see my feedback on JoinWindows > > > > > >6. UnlimitedWindows: see my feedback on JoinWindows > > > > > >7. ReadOnlyWindowStore: the existing `long` methods should be > > > > > >deprecated. (we should add `WindowStoreIterator fetch(K > key, > > > > > > long > > > > > >timeFrom, long timeTo)` to WindowStore) > > > > > >8. SessionBytesStoreSupplier: Both of those methods are > "internal > > > > > > use > > > > > >methods", so we should just leave them alone and not add new > ones. > > > > > >9. SessionStore: I don't think these are "external use" > methods > > > > > > (only > > > > > >ReadOnlySessionStore is used in IQ) maybe we should just leave > > > > > > them > > > > > > > > > > alone? > > > > > >10. Stores: I think we can just deprecate without replacement > the > > > > > > > > > > method > > > > > >that takes `segmentInterval`. > > > > > >11. WindowBytesStoreSupplier: I think this interface is also > > > > > > "internal > > > > > >use" and can be left alone > > > > > > > > > > > > Thank you for the very clear KIP that makes this discussion > > > > > > possible. In > > > > > > general, to justify some of those comments, it's easier to add > > > > > > missing > > > > > > methods later on than to remove them, so I'm erring on the side > of > > > > > > only > > > > > > adding new
kafka.tools.DumpLogSegments
Is using /opt/confluent-4.1.1/bin/kafka-run-class kafka.tools.DumpLogSegments kafka.tools.DumpLogSegments –files --print-data-log, correct way to verify that the kafka logs are compressed, if using compression.type=snappy? I have set my compression.type: snappy in my property file for kafka streams application, and when I execute using /opt/confluent-4.1.1/bin/kafka-run-class kafka.tools.DumpLogSegments kafka.tools.DumpLogSegments … I get compresscodec: SNAPPY, but the logs came out readable. If I ‘head’ or ‘tail’ .log file directly, I can see the file is compressed, it shows funny characters, indicating compressed log file. My understanding is that reading a compressed logs using consumer API, the logs are decompressed automatically without additional work in my code. Executing kafka-run-class kafka.tools.DumpLogSegments kafka.tools.DumpLogSegments, does it decompress the logs as well, or is it a direct dump of compressed logs? Meeiling Bradley
Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)
I just updated the draft implementation[^1], rebasing against the latest
trunk and implementing error routine (i.e., Error code 74 for
UnsupportedCompressionTypeException.) Since we decided to disallow all
fetch request below version 2.1.0 for the topics specifying ZStandard, I
added an error logic only.
Please have a look when you are free.
Thanks,
Dongjin
[^1]: Please check the last commit here:
https://github.com/apache/kafka/pull/2267
On Thu, Aug 23, 2018, 8:55 AM Dongjin Lee wrote:
> Jason,
>
> Great. +1 for UNSUPPORTED_COMPRESSION_TYPE.
>
> Best,
> Dongjin
>
> On Thu, Aug 23, 2018 at 8:19 AM Jason Gustafson
> wrote:
>
>> Hey Dongjin,
>>
>> Yeah that's right. For what it's worth, librdkafka also appears to handle
>> unexpected error codes. I expect that most client implementations would
>> either pass through the raw type or convert to an enum using something
>> like
>> what the java client does. Since we're expecting the client to fail
>> anyway,
>> I'm probably in favor of using the UNSUPPORTED_COMPRESSION_TYPE error
>> code.
>>
>> -Jason
>>
>> On Wed, Aug 22, 2018 at 1:46 AM, Dongjin Lee wrote:
>>
>> > Jason and Ismael,
>> >
>> > It seems like the only thing we need to regard if we define a new error
>> > code (i.e., UNSUPPORTED_COMPRESSION_TYPE) would be the implementation of
>> > the other language clients, right? At least, this strategy causes any
>> > problem for Java client. Do I understand correctly?
>> >
>> > Thanks,
>> > Dongjin
>> >
>> > On Wed, Aug 22, 2018 at 5:43 PM Dongjin Lee wrote:
>> >
>> > > Jason,
>> > >
>> > > > I think we would only use this error code when we /know/ that zstd
>> was
>> > > in use and the client doesn't support it? This is true if either 1)
>> the
>> > > message needs down-conversion and we encounter a zstd compressed
>> message,
>> > > or 2) if the topic is explicitly configured to use zstd.
>> > >
>> > > Yes, it is right. And you know, the case 1 includes 1.a) old clients'
>> > > request v0, v1 records or 1.b) implicit zstd, the compression type of
>> > > "producer" with Zstd compressed data.
>> > >
>> > > > However, if the compression type is set to "producer," then the
>> fetched
>> > > data may or may not be compressed with zstd. In this case, we return
>> the
>> > > data to the client and expect it to fail parsing. Is that correct?
>> > >
>> > > Exactly.
>> > >
>> > > Following your message, I reviewed the implementation of
>> > > `KafkaApis#handleFetchRequest,` which handles the fetch request. And
>> > found
>> > > that the information we can use is like the following:
>> > >
>> > > 1. Client's fetch request version. (`versionId` variable)
>> > > 2. Log's compression type. (`logConfig` variable)
>> > >
>> > > We can't detect the actual compression type of the data, unless we
>> > inspect
>> > > the `RecordBatch` included in the `Records` instance (i.e.,
>> > > `unconvertedRecords` variable.) Since it requires some performance
>> issue,
>> > > it is not our option - in short, we can't be sure if given chunks of
>> data
>> > > are compressed with zstd or not.
>> > >
>> > > So, conclusion: we can return an error in the case of 1.a and 2
>> easily,
>> > > with the information above. In the case 1.b (implicit zstd), we can
>> just
>> > > return the data by do nothing special and expect it to fail parsing.
>> > >
>> > > Thanks,
>> > > Dongjin
>> > >
>> > > On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma
>> wrote:
>> > >
>> > >> Jason, that's an interesting point regarding the Java client. Do we
>> know
>> > >> what clients in other languages do in these cases?
>> > >>
>> > >> Ismael
>> > >>
>> > >> On Tue, 21 Aug 2018, 17:30 Jason Gustafson,
>> wrote:
>> > >>
>> > >> > Hi Dongjin,
>> > >> >
>> > >> > One of the complications is that old versions of the API will not
>> > >> expect a
>> > >> > new error code. However, since we expect this to be a fatal error
>> > anyway
>> > >> > for old clients, it may still be more useful to return the correct
>> > error
>> > >> > code. For example, the Kafka clients use the following code to
>> convert
>> > >> the
>> > >> > error code:
>> > >> >
>> > >> > public static Errors forCode(short code) {
>> > >> > Errors error = codeToError.get(code);
>> > >> > if (error != null) {
>> > >> > return error;
>> > >> > } else {
>> > >> > log.warn("Unexpected error code: {}.", code);
>> > >> > return UNKNOWN_SERVER_ERROR;
>> > >> > }
>> > >> > }
>> > >> >
>> > >> > If we return an unsupported error code, it will be converted to an
>> > >> UNKNOWN
>> > >> > error, but at least we will get the message in the log with the
>> > correct
>> > >> > code. That seems preferable to returning a misleading error code.
>> So I
>> > >> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error
>> even
>> > for
>> > >> > older versions.
>> > >> >
>> > >> > Also, one question just to check my understanding. I think we would
>> > only
>> > >> > use this error code
Re: [DISCUSS] KIP-363: Allow performance tools to print final results to output file
Hi all, I have updated the KIP based on the suggestions received so far, please take a look at the new version: https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Allow+performance+tools+to+print+final+results+to+output+file Best, - Attila On Fri, Aug 24, 2018 at 3:52 PM Attila Sasvári wrote: > Thanks for your feedback, Kevin & Viktor! I will update the KIP next week. > > @Kevin - I also had to write sed based one-liners when I ran performance > tests to extract results (that were displayed in some charts later). > 1. Initially, I wanted to overwrite any exisisting file and document this > behaviour in the help message. Throwing an exception might be a better > idea. > 2. Yes, tools shall continue to print out all the information to standard > output/error. Specifying the output-path option will only cause that the > final results (last line about the performance metrics) will be printed to > the given file. > > @ Viktor > 1. Thanks, I will add an example. > 2. I would suggest the use of Apache Commons CSV library > (https://commons.apache.org/proper/commons-csv/) to avoid reinventing the > wheel. It is an implementation detail, so I would not add this to the KIP. > 3. Delimiters/separators shall not be configurable (to keep things simple, > only CSV is supported). Fields within a record are separated by commas; > records are separated by line break(s). More info about the CSV format: > https://tools.ietf.org/html/rfc4180#section-1 > > Best, > - Attila > > Viktor Somogyi-Vass (időpont: 2018. aug. 24., > P, 14:27) ezt írta: > >> Hi Attila, >> >> Thanks for the KIP, I think overall it looks good. I have three comments: >> 1. Would you mind adding an example? (Later on we'd need anyway for the >> public doc.) >> 2. Do you want to add any 3rd party CSV reader/writer library or will you >> implement that too? >> 3. What is the separator or is that configurable? >> >> Cheers, >> Viktor >> >> On Fri, Aug 24, 2018 at 8:18 AM Kevin Lu wrote: >> >> > Hi Attila, >> > >> > Thanks for the KIP. >> > >> > I think this would be a useful feature. Every time I have to benchmark >> > using these performance tools, I end up redirecting the output to a file >> > anyways. >> > >> > Just a couple minor questions... >> > >> > 1. If the configured file already exists, what would be the outcome? My >> > intuition is that the performance tool will spit out some type of error >> and >> > quit as we do not want to accidentally overwrite files. >> > >> > 2. Will the performance tool still output directly to shell if this >> option >> > is specified? >> > >> > Regards, >> > Kevin >> > >> > On Wed, Aug 22, 2018 at 12:16 PM Attila Sasvári >> > wrote: >> > >> > > Hi all, >> > > >> > > I have created a minor KIP to allow consumer and producer performance >> > tools >> > > to print final results to output file in CSV format. >> > > >> > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Allow+performance+tools+to+print+final+results+to+output+file >> > > >> > > Please take a look and share your thoughts! >> > > >> > > Thanks, >> > > Attila >> > > >> > >> >
Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection
Yes I’m more than happy to change it to a more appropriate name. The issue with RoundRobinPatitoner is that the DefaultPartitioner already has a Round-Robin associated to it. But if community doesn’t mind the name, I don’t either. Thanks for reading the KIP btw. Regards, On Fri, 31 Aug 2018 at 05:47, Magesh Nandakumar wrote: > +1 for this. The only small suggestion would be to possibly call this > RondRobinPartitioner which makes the intent obvious. > > On Thu, Aug 30, 2018 at 5:31 PM Stephen Powis > wrote: > > > Neat, this would be super helpful! I submitted this ages ago: > > https://issues.apache.org/jira/browse/KAFKA- > > > > On Fri, Aug 31, 2018 at 5:04 AM, Satish Duggana < > satish.dugg...@gmail.com> > > wrote: > > > > > +including both dev and user mailing lists. > > > > > > Hi, > > > Thanks for the KIP. > > > > > > "* For us, the message keys represent some metadata which we use to > > either > > > ignore messages (if a loop-back to the sender), or log some > > information.*" > > > > > > Above statement was mentioned in the KIP about how key value is used. I > > > guess the topic is not configured to be compacted and you do not want > to > > > have partitioning based on that key. IMHO, it qualifies more as a > header > > > than a key. What do you think about building records with a specific > > header > > > and consumers to execute the logic whether to process or ignore the > > > messages based on that header value. > > > > > > Thanks, > > > Satish. > > > > > > > > > On Fri, Aug 31, 2018 at 1:32 AM, Satish Duggana < > > satish.dugg...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > Thanks for the KIP. > > > > > > > > "* For us, the message keys represent some metadata which we use to > > > > either ignore messages (if a loop-back to the sender), or log some > > > > information.*" > > > > > > > > Above statement was mentioned in the KIP about how key value is > used. I > > > > guess the topic is not configured to be compacted and you do not want > > to > > > > have partitioning based on that key. IMHO, it qualifies more as a > > header > > > > than a key. What do you think about building records with a specific > > > header > > > > and consumers to execute the logic whether to process or ignore the > > > > messages based on that header value. > > > > > > > > Thanks, > > > > Satish. > > > > > > > > > > > > On Fri, Aug 31, 2018 at 12:02 AM, M. Manna > wrote: > > > > > > > >> Hi Harsha, > > > >> > > > >> thanks for reading the KIP. > > > >> > > > >> The intent is to use the DefaultPartitioner logic for round-robin > > > >> selection > > > >> of partition regardless of key type. > > > >> > > > >> Implementing Partitioner interface isn’t the issue here, you would > > have > > > to > > > >> do that anyway if you are implementing your own. But we also want > > this > > > to > > > >> be part of formal codebase. > > > >> > > > >> Regards, > > > >> > > > >> On Thu, 30 Aug 2018 at 16:58, Harsha wrote: > > > >> > > > >> > Hi, > > > >> > Thanks for the KIP. I am trying to understand the intent of > > the > > > >> > KIP. Is the use case you specified can't be achieved by > > implementing > > > >> the > > > >> > Partitioner interface here? > > > >> > https://github.com/apache/kafka/blob/trunk/clients/src/main/ > > > >> java/org/apache/kafka/clients/producer/Partitioner.java#L28 > > > >> > . > > > >> > Use your custom partitioner to be configured in your producer > > clients. > > > >> > > > > >> > Thanks, > > > >> > Harsha > > > >> > > > > >> > On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote: > > > >> > > Hello, > > > >> > > > > > >> > > I opened a very simple KIP and there exists a JIRA for it. > > > >> > > > > > >> > > I would be grateful if any comments are available for action. > > > >> > > > > > >> > > Regards, > > > >> > > > > >> > > > > > > > > > > > > > >
[jira] [Created] (KAFKA-7365) max.poll.records setting in Kafka Consumer is not working
Kashyap Ivaturi created KAFKA-7365: -- Summary: max.poll.records setting in Kafka Consumer is not working Key: KAFKA-7365 URL: https://issues.apache.org/jira/browse/KAFKA-7365 Project: Kafka Issue Type: Bug Components: consumer Reporter: Kashyap Ivaturi Hi, I have a requirement where I consume messages one by one, each message has additional processing that I should do and then manually commit the offset. Things work well most of the times until I get a big bunch of records which takes longer time to process and I encounter CommitFailed exception for the last set of records even though they were processed. While i'am able to reconnect back its picking some messages that I had already processed. I don't want this to happen as its creating duplicates in target systems that I integrate with while processing the message. I decided that even though there are more messages in the queue , I would like to have a control on how many records I can process when polled. I tried to replicate a scenario where I have started the consumer by setting 'max.poll.records' to '1' and then pushed 4 messages into the Topic the consumer is listening. I expected that the consumer will only process 1 message because of my 'max.poll.records' setting but the consumer has processed all the 4 messages in single poll. Any idea why did it not consider 'max.poll.records' setting or is some other setting overriding this setting?. Appreciate your help or guidance in troubleshooting this issue. Here is the log of my Consumer config when it starts: 2018-08-28 08:29:47.873 INFO 91121 --- [ main] o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values: [auto.commit.interval.ms|https://auto.commit.interval.ms/] = 5000 auto.offset.reset = earliest bootstrap.servers = [messaging-rtp3.cisco.com:9093] check.crcs = true [client.id|https://client.id/] = [connections.max.idle.ms|https://connections.max.idle.ms/] = 54 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 [fetch.max.wait.ms|https://fetch.max.wait.ms/] = 500 fetch.min.bytes = 1 [group.id|https://group.id/] = empestor [heartbeat.interval.ms|https://heartbeat.interval.ms/] = 3000 interceptor.classes = null internal.leave.group.on.close = true isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer max.partition.fetch.bytes = 1048576 [max.poll.interval.ms|https://max.poll.interval.ms/] = 30 max.poll.records = 1 [metadata.max.age.ms|https://metadata.max.age.ms/] = 30 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO [metrics.sample.window.ms|https://metrics.sample.window.ms/] = 3 partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 [reconnect.backoff.max.ms|https://reconnect.backoff.max.ms/] = 1000 [reconnect.backoff.ms|https://reconnect.backoff.ms/] = 50 [request.timeout.ms|https://request.timeout.ms/] = 4 [retry.backoff.ms|https://retry.backoff.ms/] = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 6 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = SSL send.buffer.bytes = 131072 [session.timeout.ms|https://session.timeout.ms/] = 1 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = [hidden] ssl.keymanager.algorithm = SunX509 ssl.keystore.location = /kafka/certs/empestor/certificates/kafka.client.empestor.keystore.jks ssl.keystore.password = [hidden] ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = /kafka/certs/empestor/certificates/kafka.client.truststore.jks ssl.truststore.password = [hidden] ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer 2018-08-28 08:29:48.079 INFO 91121 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version : 1.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] KIP-358: Migrate Streams API to Duration instead of long ms times
Hello, Bill > In the "Proposed Changes" section, there is "Try to reduce the visibility of > methods in next tickets" does that mean eventual deprecation and removal? 1. Some methods will become deprecated. I think they will be removed in the future. You can find list of deprecated methods in KIP. 2. Some internal methods can't be deprecated or hid from the user for now. I was trying to say that we should research possibility to reduce visibility of *internal* methods that are *public* now. That kind of changes is out of the scope of current KIP, so we have to do it in the next tickets. I don't expect that internal methods will be removed. В Чт, 30/08/2018 в 18:59 -0400, Bill Bejeck пишет: > Sorry for chiming in late, there was a lot of detail to catch up on. > > Overall I'm +1 in the KIP. But I do have one question about the KIP in > regards to Matthias's comments about defining dual use. > > In the "Proposed Changes" section, there is "Try to reduce the visibility > of methods in next tickets" does that mean eventual deprecation and removal? > I thought we were aiming to keep the dual use methods? Or does that imply > we'll strive for more clear delineation between DSL and internal use? > > Thanks, > Bill > > > > On Thu, Aug 30, 2018 at 5:59 PM Nikolay Izhikov wrote: > > > John, thank you. > > > > I've updated KIP. > > > > > > Dear commiters, please take a look and share your opinion. > > > > В Чт, 30/08/2018 в 14:58 -0500, John Roesler пишет: > > > Oh! I missed one minor thing: UnlimitedWindows doesn't need to set grace > > > (it currently does not either). > > > > > > Otherwise, it looks good to me! > > > > > > Thanks so much, > > > -John > > > > > > On Thu, Aug 30, 2018 at 5:30 AM Nikolay Izhikov > > > > wrote: > > > > > > > Hello, John. > > > > > > > > I've updated KIP according on your comments. > > > > Please, take a look. > > > > > > > > Are we ready to vot now? > > > > > > > > В Ср, 29/08/2018 в 14:51 -0500, John Roesler пишет: > > > > > Hey Nikolay, sorry for the silence. I'm taking another look at the > > > > KIP > > > > > before voting... > > > > > > > > > > > > > > >1. I think the Window constructor should actually be protected. I > > > > > > > > don't > > > > >know if we need a constructor that takes Instant, but if we do add > > > > > > > > one, it > > > > >should definitely be protected. > > > > >2. `long JoinWindows#size()` is overridden from `long > > > > Windows#size()`, > > > > >and should not be deprecated. Also, I don't think we need a > > > > `Duration > > > > >JoinWindows#windowSize()`. > > > > >3. Likewise, `JoinWindows#windowsFor()` is overridden from > > > > >`Windows#windowsFor()` and should also not be deprecated, and we > > > > also > > > > > > > > don't > > > > >need a `Map windowsForTime(final Instant > > > > timestamp)` > > > > >version. > > > > >4. TimeWindowedDeserializer is a bit of a puzzle for me. It > > > > actually > > > > >looks like it's incorrectly implemented! I'm not sure if we > > > > want/need > > > > > > > > to > > > > >update any of its methods or constructors. > > > > >5. TimeWindows: see my feedback on JoinWindows > > > > >6. UnlimitedWindows: see my feedback on JoinWindows > > > > >7. ReadOnlyWindowStore: the existing `long` methods should be > > > > >deprecated. (we should add `WindowStoreIterator fetch(K key, > > > > long > > > > >timeFrom, long timeTo)` to WindowStore) > > > > >8. SessionBytesStoreSupplier: Both of those methods are "internal > > > > use > > > > >methods", so we should just leave them alone and not add new ones. > > > > >9. SessionStore: I don't think these are "external use" methods > > > > (only > > > > >ReadOnlySessionStore is used in IQ) maybe we should just leave > > > > them > > > > > > > > alone? > > > > >10. Stores: I think we can just deprecate without replacement the > > > > > > > > method > > > > >that takes `segmentInterval`. > > > > >11. WindowBytesStoreSupplier: I think this interface is also > > > > "internal > > > > >use" and can be left alone > > > > > > > > > > Thank you for the very clear KIP that makes this discussion > > > > possible. In > > > > > general, to justify some of those comments, it's easier to add > > > > missing > > > > > methods later on than to remove them, so I'm erring on the side of > > > > only > > > > > adding new variants when they show up in DSL code, not worrying > > > > about the > > > > > lower-level APIs. > > > > > > > > > > What do you think about this? > > > > > -John > > > > > > > > > > On Wed, Aug 29, 2018 at 11:14 AM Nikolay Izhikov < > > > > nizhi...@apache.org> > > > > > wrote: > > > > > > > > > > > Hello, All. > > > > > > > > > > > > Calling a vote on KIP-358 [1] > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > >
[jira] [Created] (KAFKA-7364) kafka periodically run into high cpu usage with ssl writing
Yu Yang created KAFKA-7364:
--
Summary: kafka periodically run into high cpu usage with ssl
writing
Key: KAFKA-7364
URL: https://issues.apache.org/jira/browse/KAFKA-7364
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.0.0
Reporter: Yu Yang
while testing ssl writing to kafka, we found that kafka often run into high cpu
usage due to inefficiency in jdk ssl implementation.
In detail, we use a test cluster that have 12 d2.8xlarge instances, jdk-10.0.2,
and hosts only one topic that have ~20k producers write to through ssl
channel. We observed that the network threads often get 100% cpu usage after
enabling ssl writing to kafka. To improve kafka's throughput, we have
"num.network.threads=32" for the broker. Even with 32 network threads, we see
the broker cpu usage jump right after ssl writing is enabled.
!Screen Shot 2018-08-30 at 10.57.32 PM.png!
When the broker's cpu usage is high, 'perf top' shows that kafka is busy with
executing code in libsunec.so. The following is a sample stack track that we
get when the broker's cpu usage was high.
{code}
Thread 77562: (state = IN_NATIVE)
- sun.security.ec.ECDSASignature.verifySignedDigest(byte[], byte[], byte[],
byte[]) @bci=0 (Compiled frame; information may be imprecise)
- sun.security.ec.ECDSASignature.engineVerify(byte[]) @bci=70, line=321
(Compiled frame)
- java.security.Signature$Delegate.engineVerify(byte[]) @bci=9, line=1222
(Compiled frame)
- java.security.Signature.verify(byte[]) @bci=10, line=655 (Compiled frame)
- sun.security.x509.X509CertImpl.verify(java.security.PublicKey,
java.lang.String) @bci=136, line=444 (Compiled frame)
-
sun.security.provider.certpath.BasicChecker.verifySignature(java.security.cert.X509Certificate)
@bci=48, line=166 (Compiled frame)
-
sun.security.provider.certpath.BasicChecker.check(java.security.cert.Certificate,
java.util.Collection) @bci=24, line=147 (Compiled frame)
-
sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(java.security.cert.CertPath,
java.util.List, java.util.List) @bci=316, line=125 (Compiled frame)
-
sun.security.provider.certpath.PKIXCertPathValidator.validate(java.security.cert.TrustAnchor,
sun.security.provider.certpath.PKIX$ValidatorParams) @bci=390, line=233
(Compiled frame)
-
sun.security.provider.certpath.PKIXCertPathValidator.validate(sun.security.provider.certpath.PKIX$ValidatorParams)
@bci=217, line=141 (Compiled frame)
-
sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(java.security.cert.CertPath,
java.security.cert.CertPathParameters) @bci=7, line=80 (Compiled frame)
- java.security.cert.CertPathValidator.validate(java.security.cert.CertPath,
java.security.cert.CertPathParameters) @bci=6, line=292 (Compiled frame)
-
sun.security.validator.PKIXValidator.doValidate(java.security.cert.X509Certificate[],
java.security.cert.PKIXBuilderParameters) @bci=34, line=357 (Compiled frame)
-
sun.security.validator.PKIXValidator.engineValidate(java.security.cert.X509Certificate[],
java.util.Collection, java.security.AlgorithmConstraints, java.lang.Object)
@bci=232, line=259 (Compiled frame)
-
sun.security.validator.Validator.validate(java.security.cert.X509Certificate[],
java.util.Collection, java.security.AlgorithmConstraints, java.lang.Object)
@bci=6, line=260 (Compiled frame)
-
sun.security.ssl.X509TrustManagerImpl.validate(sun.security.validator.Validator,
java.security.cert.X509Certificate[], java.security.AlgorithmConstraints,
java.lang.String) @bci=10, line=324 (Compiled frame)
-
sun.security.ssl.X509TrustManagerImpl.checkTrusted(java.security.cert.X509Certificate[],
java.lang.String, javax.net.ssl.SSLEngine, boolean) @bci=179, line=279
(Compiled frame)
-
sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(java.security.cert.X509Certificate[],
java.lang.String, javax.net.ssl.SSLEngine) @bci=5, line=130 (Compiled frame)
-
sun.security.ssl.ServerHandshaker.clientCertificate(sun.security.ssl.HandshakeMessage$CertificateMsg)
@bci=190, line=1966 (Compiled frame)
- sun.security.ssl.ServerHandshaker.processMessage(byte, int) @bci=160,
line=237 (Compiled frame)
- sun.security.ssl.Handshaker.processLoop() @bci=96, line=1052 (Compiled frame)
- sun.security.ssl.Handshaker$1.run() @bci=4, line=992 (Compiled frame)
- sun.security.ssl.Handshaker$1.run() @bci=1, line=989 (Compiled frame)
-
java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
java.security.AccessControlContext) @bci=0 (Compiled frame)
- sun.security.ssl.Handshaker$DelegatedTask.run() @bci=24, line=1467 (Compiled
frame)
- org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks()
@bci=13, line=393 (Compiled frame)
- org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(boolean)
@bci=88, line=473 (Compiled frame)
-
[jira] [Created] (KAFKA-7363) How num.stream.threads in streaming application influence memory consumption?
Seweryn Habdank-Wojewodzki created KAFKA-7363: - Summary: How num.stream.threads in streaming application influence memory consumption? Key: KAFKA-7363 URL: https://issues.apache.org/jira/browse/KAFKA-7363 Project: Kafka Issue Type: Task Reporter: Seweryn Habdank-Wojewodzki Dears, How option _num.stream.threads_ in streaming application influence memory consumption? I see that by increasing num.stream.threads my application needs more memory. This is obvious, but it is not obvious how much I need to give it. Try and error method does not work, as it seems to be highly dependen on forced throughput. I mean: higher load more memory is needed. Thanks for help and regards, Seweryn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
