Hi all,
Following the upgrade to 3.1.1, I see a couple of issues.
Spark Structured Streaming (SSS) does not seem to work with the newer
spark-sql-kafka-0-10_2.12-3.1.1.jar for Spark. It throws
java.lang.NoSuchMethodError:
org.apache.spark.kafka010.KafkaTokenUtil$.needTokenUpdate(Ljava/util/Map;
Many thanks Muru. That was a great help!
-
---+-+---+
|key |value
|headers|
+
You should include commons-pool2-2.9.0.jar and remove
spark-streaming-kafka-0-10_2.12-3.0.1.jar (unnecessary jar).
On Mon, Feb 22, 2021 at 12:42 PM Mich Talebzadeh
wrote:
> Hi,
>
> Trying to make PySpark with PyCharm work with Structured Streaming
>
> spark-3.0.1-bin-hadoop3.2
> kafka_2.12-1.1.0
Hi,
Trying to make PySpark with PyCharm work with Structured Streaming
spark-3.0.1-bin-hadoop3.2
kafka_2.12-1.1.0
Basic code
from __future__ import print_function
from src.config import config, hive_url
import sys
from sparkutils import sparkstuff as s
class MDStreaming:
def __init__(self,
