Re: Mirror maker worker can't issue with REST uri
Anup, This is the expected behavior of the MirrorMaker2 application when a connector attempts to reconfigure it's tasks. It is a limitation of the MirrorMaker2 distributed mode, and has an improvement in-progress that I don't believe has been released yet. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters for more details. As a workaround, I believe you can restart the MirrorMaker2 connectors to force a reconfiguration. I hope this helps, Greg Harris On Tue, Feb 7, 2023, 10:47 PM Shirolkar, Anup wrote: > Hi, > > I have deployed a 3-node mirror maker cluster version 3.2.1 > I have configured the connect-mirror-maker.properties file and started the > mirror service using connect-mirror-maker.sh > > It runs fine but one of the three workers always gets below exception. > If I restart the connect worker with the error, another worker gets the > same error. > > > [2023-02-08 06:11:03,509] ERROR [Worker clientId=connect-2, > groupId=ruh-mm2] Request to leader to reconfigure connector tasks failed > (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1610) > org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Error > trying to forward REST request: Invalid URI host: null (authority: null) > at > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:147) > at > org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.lang.IllegalArgumentException: Invalid URI host: null > (authority: null) > at > org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:521) > at > org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:506) > at > org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:464) > at > org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:453) > at > org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:107) > ... 6 more > > What could be wrong here can you please advise. > > > Thanks, > Anup Shirolkar. > > > >
Mirror maker worker can't issue with REST uri
Hi, I have deployed a 3-node mirror maker cluster version 3.2.1 I have configured the connect-mirror-maker.properties file and started the mirror service using connect-mirror-maker.sh It runs fine but one of the three workers always gets below exception. If I restart the connect worker with the error, another worker gets the same error. [2023-02-08 06:11:03,509] ERROR [Worker clientId=connect-2, groupId=ruh-mm2] Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1610) org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Error trying to forward REST request: Invalid URI host: null (authority: null) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:147) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$reconfigureConnector$32(DistributedHerder.java:1607) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.IllegalArgumentException: Invalid URI host: null (authority: null) at org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:521) at org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:506) at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:464) at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:453) at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:107) ... 6 more What could be wrong here can you please advise. Thanks, Anup Shirolkar.
CVE-2023-25194: Apache Kafka: Possible RCE/Denial of service attack via SASL JAAS JndiLoginModule configuration using Kafka Connect
Severity: important
Description:
A possible security vulnerability has been identified in Apache Kafka
Connect. This requires access to a Kafka Connect worker,
and the ability to create/modify connectors on it with an arbitrary
Kafka client SASL JAAS config and a SASL-based security protocol,
which has been possible on Kafka Connect clusters since Apache Kafka
2.3.0. When configuring the connector via the Kafka Connect REST API,
an authenticated operator can set the `sasl.jaas.config` property for any
of the connector's Kafka clients to
"com.sun.security.auth.module.JndiLoginModule",
which can be done via the `producer.override.sasl.jaas.config`,
`consumer.override.sasl.jaas.config`, or
`admin.override.sasl.jaas.config` properties.
This will allow the server to connect to the attacker's LDAP server
and deserialize the LDAP response, which the attacker can use to
execute java deserialization gadget chains on the Kafka connect
server. Attackers can cause unrestricted deserialization of untrusted
data (or) RCE vulnerability when there are gadgets in the classpath.
Since Apache Kafka 3.0.0, users are allowed to specify these properties
in connector configurations for Kafka Connect clusters running with
out-of-the-box configurations. Before Apache Kafka 3.0.0, users may not
specify these properties unless the Kafka Connect cluster has been reconfigured
with a connector client override policy that permits them.
Since Apache Kafka 3.4.0, we have added a system property
("-Dorg.apache.kafka.disallowed.login.modules") to disable the
problematic login modules usage in SASL JAAS configuration. Also by
default "com.sun.security.auth.module.JndiLoginModule" is disabled
in Apache Kafka 3.4.0.
We advise the Kafka Connect users to validate connector configurations
and only allow trusted JNDI configurations. Also examine connector
dependencies for vulnerable versions and either upgrade their
connectors, upgrading that specific dependency, or removing the
connectors as options for remediation. Finally, in addition to leveraging the
"org.apache.kafka.disallowed.login.modules" system property, Kafka Connect users
can also implement their own connector client config override policy, which can
be used to control which Kafka client properties can be overridden directly
in a connector config and which cannot.
Credit:
Apache Kafka would like to thank Jari Jääskelä
(https://hackerone.com/reports/1529790)
and 4ra1n and Y4tacker (they found vulnerabilities in other Apache projects.
After discussion between PMC of the two projects, it was finally
confirmed that it was the vulnerability of Kafka then they reported it to us)
References:
https://kafka.apache.org/cve-list
https://kafka.apache.org/
https://www.cve.org/CVERecord?id=CVE-2023-25194
[ANNOUNCE] Apache Kafka 3.4.0
The Apache Kafka community is pleased to announce the release of Apache Kafka 3.4.0. This is a major release and it includes fixes and improvements from over 120 JIRAs. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.4.0/RELEASE_NOTES.html An overview of the release can be found in our announcement blog post: https://blogs.apache.org/kafka/entry/what-s-new-in-apache9 You can download the source and binary release (Scala 2.11 and Scala 2.12) from: https://kafka.apache.org/downloads#3.4.0 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you to the following 117 contributors to this release! A. Sophie Blee-Goldman, Ahmed Sobeh, Akhilesh C, Akhilesh Chaganti, Alan Sheinberg, aLeX, Alex Sorokoumov, Alexandre Garnier, Alyssa Huang, Andras Katona, Andrew Borley, Andrew Dean, andymg3, Artem Livshits, Ashmeet Lamba, Badai Aqrandista, Bill Bejeck, Bruno Cadonna, Calvin Liu, Chase Thomas, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Christopher L. Shannon, Colin P. McCabe, Colin Patrick McCabe, Dalibor Plavcic, Dan Stelljes, Daniel Fonai, David Arthur, David Jacot, David Karlsson, David Mao, dengziming, Derek Troy-West, Divij Vaidya, Edoardo Comar, Elkhan Eminov, Eugene Tolbakov, Federico Valeri, Francesco Nigro, FUNKYE, Greg Harris, Guozhang Wang, Hao Li, Himani Arora, Huilin Shi, Igor Soarez, Ismael Juma, James Hughes, Janik Dotzel, Jason Gustafson, Jeff Kim, Jim Galasyn, JK-Wang, Joel Hamill, John Roesler, Jonathan Albrecht, Jordan Bull, Jorge Esteban Quilcate Otoya, José Armando García Sancio, Justine Olshan, K8sCat, Kirk True, Kvicii, Levani Kokhreidze, Liam Clarke-Hutchinson, LinShunKang, liuzc9, liuzhuang2017, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Matthew de Detrich, Matthew Stidham, Matthias J. Sax, Mickael Maison, Nandini Anagondi, Nick Telford, nicolasguyomar, Niket, Niket Goel, Nikolay, Okada Haruki, Oliver Eikemeier, Omnia G H Ibrahim, Orsák Maroš, Patrik Marton, Peter Nied, Philip Nee, Philipp Trulson, Pratim SC, Proven Provenzano, Purshotam Chauhan, Rajini Sivaram, Ramesh, Rens Groothuijsen, RivenSun, Rohan, Ron Dagostino, runom, Sanjana Kaundinya, Satish Duggana, Shawn, Shay Lin, Shenglong Zhang, srishti-saraswat, Stanislav Vodetskyi, Sushant Mahajan, Tom Bentley, vamossagar12, venkatteki, Vicky Papavasileiou, Walker Carlson, Yash Mayya, zou shengfu, 行路难行路 We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/ Thank you! Regards, David Arthur
RequestsPerSec version
Hi! We have been trying to figure out why we see a high value of ProduceMessageConversionsPerSec and potentially high CPU usage. I was trying to understand what version our producers/consumers were using, but I was unable to grok this. Any help would be appreciated. We are seeing values of `version=8` and `version=9` for RequestsPerSec "request=produce", and we found: https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L163-L171 which indicate version 3.4 and 3.5. Im following what is described in KIP-272. Although I later found KIP-511 which seems to expose a new metric, KIP-896 also implies that versions are checked through the RequestsPerSec metric. Any guidance will be appreciated. References: - https://cwiki.apache.org/confluence/display/KAFKA/KIP-272%3A+Add+API+version+tag+to+broker%27s+RequestsPerSec+metric - https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers - https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 Thanks Gonzalo
Kafka older topic reassign to newly created broker!
Hi Team, Need help to reassign existing topics to newly added brokers. Earlier we had 3 brokers and 1 zookeeper. Now we have added one more broker. We are trying to move our already some of existing topics to assign on new broker, partition is happening but while reassign, its going in still in progress(pending) for infinite. Can you guys help me with this? kafka version: 2.1.1 Followed Docs: https://stackoverflow.com/questions/74010964/add-a-new-partition-to-existing-topic-and-assign-it-to-a-specific-node https://medium.com/@abdullahtrmn/reassigning-kafka-partitions-7f9ae0989317
Kafka Mirror maker stops replicating
Hi, Hope this is the right forum to ask for Kafka mirror maker issues. We are facing an issue where the mirror maker replicates the trades and then doesn't work for long time and again replicates. Also seeing the warning message to increase the poll interval or decrease the maximum batch size (max.poll.records). I have tried reducing max.poll.records to 250 but still same issues. Could anyone suggest what could be wrong? Thanks
