[jira] [Issue Comment Deleted] (KAFKA-7028) super.users doesn't work with custom principals
[
https://issues.apache.org/jira/browse/KAFKA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KAFKA-7028:
--
Comment: was deleted
(was: {code}
superUsers = configs.get(SimpleAclAuthorizer.SuperUsersProp).collect {
case str: String if str.nonEmpty => str.split(";").map(s =>
SecurityUtils.parseKafkaPrincipal(s.trim)).toSet
}.getOrElse(Set.empty[KafkaPrincipal])
{code}
Looking at KafkaPrincipalBuilder, I haven't found the relevant method for
constructing KafkaPrincipal which is usable by the above code.)
> super.users doesn't work with custom principals
> ---
>
> Key: KAFKA-7028
> URL: https://issues.apache.org/jira/browse/KAFKA-7028
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>
> SimpleAclAuthorizer creates a KafkaPrincipal for the users defined in the
> super.users broker config. However, it should use the configured
> KafkaPrincipalBuilder so that it works with a custom defined one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507172#comment-16507172 ] rajadayalan perumalsamy commented on KAFKA-7012: Need to check regarding the correctness/missing msgs. Will check and update. > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6971) Passing in help flag to kafka-console-producer should print arg options
[
https://issues.apache.org/jira/browse/KAFKA-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507122#comment-16507122
]
ASF GitHub Bot commented on KAFKA-6971:
---
rajkrrsingh opened a new pull request #5175: KAFKA-6971: Passing in help flag
to kafka-console-producer should pri…
URL: https://github.com/apache/kafka/pull/5175
this small fix introduces to print all available option for Kafka console
producer and consumer. I did few tests while producing and consuming messages
after this one-liner change which looks positive and dont have any side effects.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Passing in help flag to kafka-console-producer should print arg options
> ---
>
> Key: KAFKA-6971
> URL: https://issues.apache.org/jira/browse/KAFKA-6971
> Project: Kafka
> Issue Type: Improvement
> Components: core, producer
>Affects Versions: 1.1.0
>Reporter: Yeva Byzek
>Priority: Minor
> Labels: newbie
>
> {{kafka-console-consumer --help}} prints "help is not a recognized option" as
> well as output of options
> {{kafka-console-producer --help}} prints "help is not a recognized option"
> but no output of options
> Possible solutions:
> (a) Enhance {{kafka-console-producer}} to also print out all options when a
> user passes in an unrecognized option
> (b) Enhance both {{kafka-console-producer}} and {{kafka-console-consumer}} to
> legitimately accept the {{--help}} flag
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer
[ https://issues.apache.org/jira/browse/KAFKA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar reassigned KAFKA-7029: Assignee: Manikumar > ReplicaVerificationTool should not use the deprecated SimpleConsumer > > > Key: KAFKA-7029 > URL: https://issues.apache.org/jira/browse/KAFKA-7029 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Assignee: Manikumar >Priority: Major > > Unless there's a reason not to, the simplest would be to use KafkaConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7030) Add configuration to disable message down-conversion
Dhruvil Shah created KAFKA-7030: --- Summary: Add configuration to disable message down-conversion Key: KAFKA-7030 URL: https://issues.apache.org/jira/browse/KAFKA-7030 Project: Kafka Issue Type: Sub-task Reporter: Dhruvil Shah Assignee: Dhruvil Shah Add configuration to disable message down-conversion as described in [KIP-283|https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer
Ismael Juma created KAFKA-7029: -- Summary: ReplicaVerificationTool should not use the deprecated SimpleConsumer Key: KAFKA-7029 URL: https://issues.apache.org/jira/browse/KAFKA-7029 Project: Kafka Issue Type: Bug Reporter: Ismael Juma Unless there's a reason not to, the simplest would be to use KafkaConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer
[ https://issues.apache.org/jira/browse/KAFKA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507081#comment-16507081 ] Ismael Juma commented on KAFKA-7029: [~omkreddy], would you be interested in picking this up? > ReplicaVerificationTool should not use the deprecated SimpleConsumer > > > Key: KAFKA-7029 > URL: https://issues.apache.org/jira/browse/KAFKA-7029 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Priority: Major > > Unless there's a reason not to, the simplest would be to use KafkaConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6953) [Streams] Schedulable KTable as Graph source (for minimizing aggregation pressure)
[ https://issues.apache.org/jira/browse/KAFKA-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506998#comment-16506998 ] Flavio Stutz commented on KAFKA-6953: - Sorry but I didn’t see your reply. Was it by email? Is Wiki ID the same as my login here in Apache Jira? If so, it is “flaviostutz” > [Streams] Schedulable KTable as Graph source (for minimizing aggregation > pressure) > -- > > Key: KAFKA-6953 > URL: https://issues.apache.org/jira/browse/KAFKA-6953 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Flavio Stutz >Priority: Major > > === PROBLEM === > We have faced the following scenario/problem in a lot of situations with > KStreams: > - Huge incoming data being processed by numerous application instances > - Need to aggregate, or count the overall data as a single value > (something like "count the total number of messages that has been processed > among all distributed instances") > - The challenge here is to manage this kind of situation without any > bottlenecks. We don't need the overall aggregation of all instances states at > each processed message, so it is possible to store the partial aggregations > on local stores and, at time to time, query those states and aggregate them, > avoiding bottlenecks. > Some ways we KNOW it wouldn't work because of bottlenecks: > - Sink all instances local counter/aggregation result to a Topic with a > single partition so that we could have another Graph with a single instance > that could aggregate all results > - In this case, if I had 500 instances processing 1000/s each (with > no bottlenecks), I would have a single partition topic with 500k messages/s > for my single aggregating instance to process that much messages (IMPOSSIBLE > bottleneck) > === TRIALS === > These are some ways we managed to do this: > - Expose a REST endpoint so that Prometheus could extract local metrics of > each application instance's state stores and them calculate the total count > on Prometheus using queries > - we don't like this much because we believe KStreams was meant to > INPUT and OUTPUT data using Kafka Topics for simplicity and power > - Create a scheduled Punctuate at the end of the Graph so that we can > query (using getAllMetadata) all other instances's state store counters, sum > them all and them publish to another Kafka Topic from time to time. > - For this to work we created a way so that only one application > instance's Punctuate algorithm would perform the calculations (something like > a master election through instance ids and metadata) > === PROPOSAL === > Create a new DSL Source with the following characteristics: > - Source parameters: "scheduled time" (using cron's like config), "state > store name", bool "from all application instances" > - Behavior: At the desired time, query all K,V tuples from the state store > and source those messages to the Graph > - If "from all application instances" is true, query the tuples > from all application instances state stores and source them all, concatenated > - This is a way to create a "timed aggregation barrier" to avoid > bottlenecks. With this we could enhance the ability of KStreams to better > handle the CAP Theorem characteristics, so that one could choose to have > Consistency over Availability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506994#comment-16506994 ] radai rosenblatt commented on KAFKA-7012: - that line is meant to handle a (hypothetical) scenario where socket data is available in the SSL cyphertext buffer but no data will ever be read from the socket itself again. i guess we'll need to add some code to only run this path every-so-often as the SSL code is apparently likely to leave things behind. did commenting this out affect function/correctness? missing msgs? higher 99th percentile latencies? > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7028) super.users doesn't work with custom principals
[
https://issues.apache.org/jira/browse/KAFKA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506993#comment-16506993
]
Ted Yu commented on KAFKA-7028:
---
{code}
superUsers = configs.get(SimpleAclAuthorizer.SuperUsersProp).collect {
case str: String if str.nonEmpty => str.split(";").map(s =>
SecurityUtils.parseKafkaPrincipal(s.trim)).toSet
}.getOrElse(Set.empty[KafkaPrincipal])
{code}
Looking at KafkaPrincipalBuilder, I haven't found the relevant method for
constructing KafkaPrincipal which is usable by the above code.
> super.users doesn't work with custom principals
> ---
>
> Key: KAFKA-7028
> URL: https://issues.apache.org/jira/browse/KAFKA-7028
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>
> SimpleAclAuthorizer creates a KafkaPrincipal for the users defined in the
> super.users broker config. However, it should use the configured
> KafkaPrincipalBuilder so that it works with a custom defined one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7028) super.users doesn't work with custom principals
[ https://issues.apache.org/jira/browse/KAFKA-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506984#comment-16506984 ] Ted Yu commented on KAFKA-7028: --- This means retrieving principalBuilder thru KafkaConfig#PrincipalBuilderClassProp , right ? > super.users doesn't work with custom principals > --- > > Key: KAFKA-7028 > URL: https://issues.apache.org/jira/browse/KAFKA-7028 > Project: Kafka > Issue Type: Bug >Reporter: Ismael Juma >Priority: Major > Fix For: 2.0.0 > > > SimpleAclAuthorizer creates a KafkaPrincipal for the users defined in the > super.users broker config. However, it should use the configured > KafkaPrincipalBuilder so that it works with a custom defined one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6574) Support Headers in console-consumer and console-producer
[ https://issues.apache.org/jira/browse/KAFKA-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506891#comment-16506891 ] Alexander Abramov commented on KAFKA-6574: -- FWIW, there is already a pull request in progress to do this for console consumer (KAFKA-6733). So this issue is partially a duplicate. > Support Headers in console-consumer and console-producer > > > Key: KAFKA-6574 > URL: https://issues.apache.org/jira/browse/KAFKA-6574 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 1.0.0 >Reporter: Vik Gamov >Priority: Minor > > Message headers were introduced in 1.0.0 but console producer and consumer > don't support them, -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506838#comment-16506838 ] rajadayalan perumalsamy commented on KAFKA-7012: yes, your hypothesis is right. cpu usage is fine with the pollSelectionKeys() line commented, screenshot attached. Tried it on 1.0.1 version also and cpu usage is fine. > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7012) Performance issue upgrading to kafka 1.0.1 or 1.1
[ https://issues.apache.org/jira/browse/KAFKA-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajadayalan perumalsamy updated KAFKA-7012: --- Attachment: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png > Performance issue upgrading to kafka 1.0.1 or 1.1 > - > > Key: KAFKA-7012 > URL: https://issues.apache.org/jira/browse/KAFKA-7012 > Project: Kafka > Issue Type: Bug >Affects Versions: 1.1.0, 1.0.1 >Reporter: rajadayalan perumalsamy >Assignee: praveen >Priority: Major > Attachments: Commit-47ee8e954-0607-bufferkeys-nopoll-profile.png, > Commit-47ee8e954-0607-memory.png, Commit-47ee8e954-0607-profile.png, > Commit-47ee8e954-profile.png, Commit-47ee8e954-profile2.png, > Commit-f15cdbc91b-profile.png, Commit-f15cdbc91b-profile2.png > > > We are trying to upgrade kafka cluster from Kafka 0.11.0.1 to Kafka 1.0.1. > After upgrading 1 node on the cluster, we notice that network threads use > most of the cpu. It is a 3 node cluster with 15k messages/sec on each node. > With Kafka 0.11.0.1 typical usage of the servers is around 50 to 60% > vcpu(using less than 1 vcpu). After upgrade we are noticing that cpu usage is > high depending on the number of network threads used. If networks threads is > set to 8, then the cpu usage is around 850%(9 vcpus) and if it is set to 4 > then the cpu usage is around 450%(5 vcpus). Using the same kafka > server.properties for both. > Did further analysis with git bisect, couple of build and deploys, traced the > issue to commit 47ee8e954df62b9a79099e944ec4be29afe046f6. CPU usage is fine > for commit f15cdbc91b240e656d9a2aeb6877e94624b21f8d. But with commit > 47ee8e954df62b9a79099e944ec4be29afe046f6 cpu usage has increased. Have > attached screenshots of profiling done with both the commits. Screenshot > Commit-f15cdbc91b-profile shows less cpu usage by network threads and > Screenshots Commit-47ee8e954-profile and Commit-47ee8e954-profile2 show > higher cpu usage(almost entire cpu usage) by network threads. Also noticed > that kafka.network.Processor.poll() method is invoked 10 times more with > commit 47ee8e954df62b9a79099e944ec4be29afe046f6. > We need the issue to be resolved to upgrade the cluster. Please let me know > if you need any additional information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
