Spark 3.2.1 is compiled vs Kafka 2.8.0; the forthcoming Spark 3.3 against Kafka 3.1.0. It may well be mutually compatible though.
On Fri, Feb 25, 2022 at 2:40 PM Michael Williams (SSI) < michael.willi...@ssigroup.com> wrote: > I believe it is 3.1, but if there is a different version that “works > better” with spark, any advice would be appreciated. Our entire team is > totally new to spark and kafka (this is a poc trial). > > > > *From:* Mich Talebzadeh [mailto:mich.talebza...@gmail.com] > *Sent:* Friday, February 25, 2022 2:30 PM > *To:* Michael Williams (SSI) <michael.willi...@ssigroup.com> > *Cc:* user@spark.apache.org > *Subject:* Re: Spark Kafka Integration > > > > and what version of kafka do you have 2.7? > > > > for spark 3.1.1 I needed these jar files to make it work > > > > kafka-clients-2.7.0.jar > commons-pool2-2.9.0.jar > spark-streaming_2.12-3.1.1.jar > spark-sql-kafka-0-10_2.12-3.1.0.jar > > > > HTH > > > > view my Linkedin profile > <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!IPetXT4!gBJNAXItekUwDwvWFDOm6TyHbSrwbIYzHfz3Lgdat86WXOH09jgo72Z1eIt0YwL6MRS0hIg$> > > > > https://en.everybodywiki.com/Mich_Talebzadeh > <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!IPetXT4!gBJNAXItekUwDwvWFDOm6TyHbSrwbIYzHfz3Lgdat86WXOH09jgo72Z1eIt0YwL6-ShVdmU$> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > > > > On Fri, 25 Feb 2022 at 20:15, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > What is the use case? Is this for spark structured streaming? > > > > HTH > > > > > > view my Linkedin profile > <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!IPetXT4!gBJNAXItekUwDwvWFDOm6TyHbSrwbIYzHfz3Lgdat86WXOH09jgo72Z1eIt0YwL6MRS0hIg$> > > > > https://en.everybodywiki.com/Mich_Talebzadeh > <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!IPetXT4!gBJNAXItekUwDwvWFDOm6TyHbSrwbIYzHfz3Lgdat86WXOH09jgo72Z1eIt0YwL6-ShVdmU$> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > > > > On Fri, 25 Feb 2022 at 19:38, Michael Williams (SSI) < > michael.willi...@ssigroup.com> wrote: > > After reviewing Spark's Kafka Integration guide, it indicates that > spark-sql-kafka-0-10_2.12_3.2.1.jar and its dependencies are needed for > Spark 3.2.1 (+ Scala 2.12) to work with Kafka. Can anybody clarify the > cleanest, most repeatable (reliable) way to acquire these jars for > including in a Spark Docker image without introducing version compatibility > issues? > > > > Thank you, > > Mike > > > > This electronic message may contain information that is Proprietary, > Confidential, or legally privileged or protected. It is intended only for > the use of the individual(s) and entity named in the message. If you are > not an intended recipient of this message, please notify the sender > immediately and delete the material from your computer. Do not deliver, > distribute or copy this message and do not disclose its contents or take > any action in reliance on the information it contains. Thank You. > > > This electronic message may contain information that is Proprietary, > Confidential, or legally privileged or protected. It is intended only for > the use of the individual(s) and entity named in the message. If you are > not an intended recipient of this message, please notify the sender > immediately and delete the material from your computer. Do not deliver, > distribute or copy this message and do not disclose its contents or take > any action in reliance on the information it contains. Thank You. >
