Re: Can the consumer know the user who sent the message ?

[Peter Bukowinski]

Kafka’s authorization layer is entirely separate from topic data, other than 
granting or denying access. If you don’t want to alter the messages themselves 
to hold information about the producers, then you should consider using 
separate topics.

-- Peter (from phone)

> On Apr 21, 2019, at 10:13 AM, Kumaran Ponnambalam  
> wrote:
> 
> Hi,
> 
> I am trying to add security to my Kafka setup. I can setup authorization for 
> a list of users. However, i would like my consumer to know about the specific 
> user who produced the message. Based on the user, i would like to perform 
> additional segmentation and processing in a multi-tenanted scenario. I don't 
> want the producer to set the user ID as a custom attribute or header, for 
> security reasons. Rather discover the user based on their authentication used.
> 
> Any thoughts on how to achieve this?
> 
> thanks
> 
> Kumaran

Re: [EXTERNAL] Re: Kafka - JMX Metrics for Fetch and Produce

[Jose Manuel Vega Monroy]

Hi there,

Included "Bandwidth quota metrics per (user, client-id), user or client-id"?

That was the metric we were monitoring, with regex to show any user and 
clientId.

Thanks
 
 
 
Jose Manuel Vega Monroy 
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.mon...@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
 
 
 

On 21/04/2019, 21:10, "Manikumar"  wrote:

Hi,

RequestsPerSec metric was changed to include the API version in Kafka 2.0
release.
You may need to update jmx query.


https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_documentation_-23upgrade-5F200-5Fnotable&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=HpbvSUu37R0CDXVebPOpAiXnWZvMUBHNRztCp9xwf20&s=zZQcoInkTwcV58vNqte10APezEUrDt-GvCevjKeN_OE&e=

https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D272-253A-2BAdd-2BAPI-2Bversion-2Btag-2Bto-2Bbroker-2527s-2BRequestsPerSec-2Bmetric&d=DwIBaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=HpbvSUu37R0CDXVebPOpAiXnWZvMUBHNRztCp9xwf20&s=s9z5arHWXznh1HdIQ4CTacRYCl885Ss84-LVkFSahro&e=

On Mon, Apr 22, 2019 at 12:23 AM Jose Manuel Vega Monroy <
jose.mon...@williamhill.com> wrote:

> Hi there,
>
>
>
> Recently I performed several * rolling upgrades following official steps*
> for our Kafka brokers *from 0.11.0.1 to newer versions in different
> environments*, and apparently working fine in all cases from functional
> point of view: *producers and consumers working as expected*.
>
>
>
> Specifically, I upgraded:
>
>
>
>1. From *0.11.0.1 to 1.0.0*, and then from *1.0.0 to 2.0.0*, and then*
>to* *2.1.1*
>2. *From 0.11.0.1 directly to 2.1.1*
>
>
>
> However, in all cases *JMX metrics for Fetch and Produce* which used to
> show *all producers and consumers working with brokers* are gone, just
> showing queue-size, in our *JMX monitoring clients* -specifically, 
*Introscope
> Wily*- keeping same configuration*.*
>
>
>
> In fact, I removed Wily filter configuration for JMX in *order to show
> all possible metrics, and keeping both Fetch and Produce still gone*.
>
>
>
> Note I checked if having proper version after rolling upgrade, for
> example, for *2.1.1*, and being as expected:
>
>
>
> *ll /opt/kafka/libs/*
>
> *total 54032*
>
> *-rw-r--r-- 1 kafka kafka69409 Jan  4 08:42 activation-1.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka14768 Jan  4 08:42
> aopalliance-repackaged-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka90347 Jan  4 08:42 argparse4j-0.7.0.jar*
>
> *-rw-r--r-- 1 kafka kafka20437 Jan  4 08:40
> audience-annotations-0.5.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   501879 Jan  4 08:43 commons-lang3-3.8.1.jar*
>
> *-rw-r--r-- 1 kafka kafka96801 Feb  8 18:32 connect-api-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka18265 Feb  8 18:32
> connect-basic-auth-extension-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka20509 Feb  8 18:32 connect-file-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka45489 Feb  8 18:32 connect-json-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   466588 Feb  8 18:32 connect-runtime-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka90358 Feb  8 18:32
> connect-transforms-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka  2442625 Jan  4 08:43 guava-20.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   186763 Jan  4 08:42 hk2-api-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   189454 Jan  4 08:42 hk2-locator-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   135317 Jan  4 08:42 hk2-utils-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka66894 Jan 11 21:28
> jackson-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   325619 Jan 11 21:27 jackson-core-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka  1347236 Jan 11 21:27 
jackson-databind-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32373 Jan 11 21:28
> jackson-jaxrs-base-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka15861 Jan 11 21:28
> jackson-jaxrs-json-provider-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32627 Jan 11 21:28
> jackson-module-jaxb-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   737884 Jan  4 08:43 javassist-3.22.0-CR2.jar*
>
> *-rw-r--r-- 1 kafka kafka26366 Jan  4 08:42
> javax.annotation-api-1.2.jar*
>
> *-rw-r--r-- 1 kafka kafka 2497 Jan  4 08:42 javax.inject-1.jar*
>
> *-rw-r--r-- 1 kafka kafka 5951 Jan  4 08:42 
javax.inject-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka95806 Jan  4 08:42
> javax.servlet-api-3.1.

Re: Kafka - JMX Metrics for Fetch and Produce

[Manikumar]

Hi,

RequestsPerSec metric was changed to include the API version in Kafka 2.0
release.
You may need to update jmx query.

http://kafka.apache.org/documentation/#upgrade_200_notable
https://cwiki.apache.org/confluence/display/KAFKA/KIP-272%3A+Add+API+version+tag+to+broker%27s+RequestsPerSec+metric

On Mon, Apr 22, 2019 at 12:23 AM Jose Manuel Vega Monroy <
jose.mon...@williamhill.com> wrote:

> Hi there,
>
>
>
> Recently I performed several * rolling upgrades following official steps*
> for our Kafka brokers *from 0.11.0.1 to newer versions in different
> environments*, and apparently working fine in all cases from functional
> point of view: *producers and consumers working as expected*.
>
>
>
> Specifically, I upgraded:
>
>
>
>1. From *0.11.0.1 to 1.0.0*, and then from *1.0.0 to 2.0.0*, and then*
>to* *2.1.1*
>2. *From 0.11.0.1 directly to 2.1.1*
>
>
>
> However, in all cases *JMX metrics for Fetch and Produce* which used to
> show *all producers and consumers working with brokers* are gone, just
> showing queue-size, in our *JMX monitoring clients* -specifically, *Introscope
> Wily*- keeping same configuration*.*
>
>
>
> In fact, I removed Wily filter configuration for JMX in *order to show
> all possible metrics, and keeping both Fetch and Produce still gone*.
>
>
>
> Note I checked if having proper version after rolling upgrade, for
> example, for *2.1.1*, and being as expected:
>
>
>
> *ll /opt/kafka/libs/*
>
> *total 54032*
>
> *-rw-r--r-- 1 kafka kafka69409 Jan  4 08:42 activation-1.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka14768 Jan  4 08:42
> aopalliance-repackaged-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka90347 Jan  4 08:42 argparse4j-0.7.0.jar*
>
> *-rw-r--r-- 1 kafka kafka20437 Jan  4 08:40
> audience-annotations-0.5.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   501879 Jan  4 08:43 commons-lang3-3.8.1.jar*
>
> *-rw-r--r-- 1 kafka kafka96801 Feb  8 18:32 connect-api-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka18265 Feb  8 18:32
> connect-basic-auth-extension-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka20509 Feb  8 18:32 connect-file-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka45489 Feb  8 18:32 connect-json-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   466588 Feb  8 18:32 connect-runtime-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka90358 Feb  8 18:32
> connect-transforms-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka  2442625 Jan  4 08:43 guava-20.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   186763 Jan  4 08:42 hk2-api-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   189454 Jan  4 08:42 hk2-locator-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka   135317 Jan  4 08:42 hk2-utils-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka66894 Jan 11 21:28
> jackson-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   325619 Jan 11 21:27 jackson-core-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka  1347236 Jan 11 21:27 jackson-databind-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32373 Jan 11 21:28
> jackson-jaxrs-base-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka15861 Jan 11 21:28
> jackson-jaxrs-json-provider-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka32627 Jan 11 21:28
> jackson-module-jaxb-annotations-2.9.8.jar*
>
> *-rw-r--r-- 1 kafka kafka   737884 Jan  4 08:43 javassist-3.22.0-CR2.jar*
>
> *-rw-r--r-- 1 kafka kafka26366 Jan  4 08:42
> javax.annotation-api-1.2.jar*
>
> *-rw-r--r-- 1 kafka kafka 2497 Jan  4 08:42 javax.inject-1.jar*
>
> *-rw-r--r-- 1 kafka kafka 5951 Jan  4 08:42 javax.inject-2.5.0-b42.jar*
>
> *-rw-r--r-- 1 kafka kafka95806 Jan  4 08:42
> javax.servlet-api-3.1.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   126898 Jan  4 08:42 javax.ws.rs-api-2.1.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   127509 Jan  4 08:42 javax.ws.rs-api-2.1.jar*
>
> *-rw-r--r-- 1 kafka kafka   125632 Jan  4 08:42 jaxb-api-2.3.0.jar*
>
> *-rw-r--r-- 1 kafka kafka   181563 Jan  4 08:42 jersey-client-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka  1140395 Jan  4 08:43 jersey-common-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka18085 Jan  4 08:42
> jersey-container-servlet-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka59332 Jan  4 08:42
> jersey-container-servlet-core-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka62547 Jan  4 08:42 jersey-hk2-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka71936 Jan  4 08:42 jersey-media-jaxb-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka   933619 Jan  4 08:43 jersey-server-2.27.jar*
>
> *-rw-r--r-- 1 kafka kafka   280573 Jan  4 08:43
> jetty-client-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka16680 Jan  4 08:42
> jetty-continuation-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   195664 Jan  4 08:42
> jetty-http-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   139436 Jan  4 08:42
> jetty-io-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka93077 Jan  4 08:42
> jetty-security-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   609290 Jan  4 08:43
> jetty-server-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 1 kafka kafka   110983 Jan  4 08:42
> jetty-servlet-9.4.12.v20180830.jar*
>
> *-rw-r--r-- 

Kafka - JMX Metrics for Fetch and Produce

[Jose Manuel Vega Monroy]

Hi there,

Recently I performed several rolling upgrades following official steps for our 
Kafka brokers from 0.11.0.1 to newer versions in different environments, and 
apparently working fine in all cases from functional point of view: producers 
and consumers working as expected.

Specifically, I upgraded:


  1.  From 0.11.0.1 to 1.0.0, and then from 1.0.0 to 2.0.0, and then to 2.1.1
  2.  From 0.11.0.1 directly to 2.1.1

However, in all cases JMX metrics for Fetch and Produce which used to show all 
producers and consumers working with brokers are gone, just showing queue-size, 
in our JMX monitoring clients -specifically, Introscope Wily- keeping same 
configuration.

In fact, I removed Wily filter configuration for JMX in order to show all 
possible metrics, and keeping both Fetch and Produce still gone.

Note I checked if having proper version after rolling upgrade, for example, for 
2.1.1, and being as expected:

ll /opt/kafka/libs/
total 54032
-rw-r--r-- 1 kafka kafka69409 Jan  4 08:42 activation-1.1.1.jar
-rw-r--r-- 1 kafka kafka14768 Jan  4 08:42 
aopalliance-repackaged-2.5.0-b42.jar
-rw-r--r-- 1 kafka kafka90347 Jan  4 08:42 argparse4j-0.7.0.jar
-rw-r--r-- 1 kafka kafka20437 Jan  4 08:40 audience-annotations-0.5.0.jar
-rw-r--r-- 1 kafka kafka   501879 Jan  4 08:43 commons-lang3-3.8.1.jar
-rw-r--r-- 1 kafka kafka96801 Feb  8 18:32 connect-api-2.1.1.jar
-rw-r--r-- 1 kafka kafka18265 Feb  8 18:32 
connect-basic-auth-extension-2.1.1.jar
-rw-r--r-- 1 kafka kafka20509 Feb  8 18:32 connect-file-2.1.1.jar
-rw-r--r-- 1 kafka kafka45489 Feb  8 18:32 connect-json-2.1.1.jar
-rw-r--r-- 1 kafka kafka   466588 Feb  8 18:32 connect-runtime-2.1.1.jar
-rw-r--r-- 1 kafka kafka90358 Feb  8 18:32 connect-transforms-2.1.1.jar
-rw-r--r-- 1 kafka kafka  2442625 Jan  4 08:43 guava-20.0.jar
-rw-r--r-- 1 kafka kafka   186763 Jan  4 08:42 hk2-api-2.5.0-b42.jar
-rw-r--r-- 1 kafka kafka   189454 Jan  4 08:42 hk2-locator-2.5.0-b42.jar
-rw-r--r-- 1 kafka kafka   135317 Jan  4 08:42 hk2-utils-2.5.0-b42.jar
-rw-r--r-- 1 kafka kafka66894 Jan 11 21:28 jackson-annotations-2.9.8.jar
-rw-r--r-- 1 kafka kafka   325619 Jan 11 21:27 jackson-core-2.9.8.jar
-rw-r--r-- 1 kafka kafka  1347236 Jan 11 21:27 jackson-databind-2.9.8.jar
-rw-r--r-- 1 kafka kafka32373 Jan 11 21:28 jackson-jaxrs-base-2.9.8.jar
-rw-r--r-- 1 kafka kafka15861 Jan 11 21:28 
jackson-jaxrs-json-provider-2.9.8.jar
-rw-r--r-- 1 kafka kafka32627 Jan 11 21:28 
jackson-module-jaxb-annotations-2.9.8.jar
-rw-r--r-- 1 kafka kafka   737884 Jan  4 08:43 javassist-3.22.0-CR2.jar
-rw-r--r-- 1 kafka kafka26366 Jan  4 08:42 javax.annotation-api-1.2.jar
-rw-r--r-- 1 kafka kafka 2497 Jan  4 08:42 javax.inject-1.jar
-rw-r--r-- 1 kafka kafka 5951 Jan  4 08:42 javax.inject-2.5.0-b42.jar
-rw-r--r-- 1 kafka kafka95806 Jan  4 08:42 javax.servlet-api-3.1.0.jar
-rw-r--r-- 1 kafka kafka   126898 Jan  4 08:42 javax.ws.rs-api-2.1.1.jar
-rw-r--r-- 1 kafka kafka   127509 Jan  4 08:42 javax.ws.rs-api-2.1.jar
-rw-r--r-- 1 kafka kafka   125632 Jan  4 08:42 jaxb-api-2.3.0.jar
-rw-r--r-- 1 kafka kafka   181563 Jan  4 08:42 jersey-client-2.27.jar
-rw-r--r-- 1 kafka kafka  1140395 Jan  4 08:43 jersey-common-2.27.jar
-rw-r--r-- 1 kafka kafka18085 Jan  4 08:42 jersey-container-servlet-2.27.jar
-rw-r--r-- 1 kafka kafka59332 Jan  4 08:42 
jersey-container-servlet-core-2.27.jar
-rw-r--r-- 1 kafka kafka62547 Jan  4 08:42 jersey-hk2-2.27.jar
-rw-r--r-- 1 kafka kafka71936 Jan  4 08:42 jersey-media-jaxb-2.27.jar
-rw-r--r-- 1 kafka kafka   933619 Jan  4 08:43 jersey-server-2.27.jar
-rw-r--r-- 1 kafka kafka   280573 Jan  4 08:43 jetty-client-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka16680 Jan  4 08:42 
jetty-continuation-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka   195664 Jan  4 08:42 jetty-http-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka   139436 Jan  4 08:42 jetty-io-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka93077 Jan  4 08:42 
jetty-security-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka   609290 Jan  4 08:43 jetty-server-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka   110983 Jan  4 08:42 
jetty-servlet-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka93161 Jan  4 08:42 
jetty-servlets-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka   502985 Jan  4 08:43 jetty-util-9.4.12.v20180830.jar
-rw-r--r-- 1 kafka kafka78146 Jan  4 08:40 jopt-simple-5.0.4.jar
-rw-r--r-- 1 kafka kafka  3949187 Feb  8 18:32 kafka_2.12-2.1.1.jar
-rw-r--r-- 1 kafka kafka  821 Feb  8 18:32 kafka_2.12-2.1.1.jar.asc
-rw-r--r-- 1 kafka kafka31034 Feb  8 18:32 kafka_2.12-2.1.1-javadoc.jar
-rw-r--r-- 1 kafka kafka  821 Feb  8 18:32 kafka_2.12-2.1.1-javadoc.jar.asc
-rw-r--r-- 1 kafka kafka10261 Feb  8 18:32 kafka_2.12-2.1.1-scaladoc.jar
-rw-r--r-- 1 kafka kafka  821 Feb  8 18:32 kafka_2.12-2.1.1-scaladoc.jar.asc
-rw-r--r-- 1 kafka kafka   728679 Feb  8 18:32 kafka_2.12-2.1.1-sources.jar
-rw-r--r-- 1 kafka kafka  821 Feb  8 1

Can the consumer know the user who sent the message ?

[Kumaran Ponnambalam]

Hi,

I am trying to add security to my Kafka setup. I can setup authorization for a 
list of users. However, i would like my consumer to know about the specific 
user who produced the message. Based on the user, i would like to perform 
additional segmentation and processing in a multi-tenanted scenario. I don't 
want the producer to set the user ID as a custom attribute or header, for 
security reasons. Rather discover the user based on their authentication used.

Any thoughts on how to achieve this?

thanks

Kumaran