Re: Spark job crashing - Spark Structured Streaming with Kafka

2021-03-03 Thread Mich Talebzadeh 
Hi Sachit,

Also ensure that you are using the correct Jar files for Spark Structured
streaming with Kafka. PLease see my mail on it.


HTH


LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*





*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 3 Mar 2021 at 10:20, Jungtaek Lim 
wrote:

> This looks to be coming from k8s, as WatchConnectionManager looks to be a
> part of kubernetes-client. If that is intermittent, you may want to check
> the health of k8s API server or network issues as well. Another bet would
> be trying out 3.1.1 as Spark announces GA on supporting k8s from this
> version.
>
> On Tue, Mar 2, 2021 at 7:08 PM Sachit Murarka 
> wrote:
>
>> Hi Jungtaek,
>>
>> Please find full logs:
>>
>> java.io.EOFException
>> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
>> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
>> at
>> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
>> at
>> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
>> at
>> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
>> at
>> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
>> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
>> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> 21/03/02 06:40:31 WARN WatchConnectionManager: Exec Failure
>> java.io.EOFException
>> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
>> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
>> at
>> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
>> at
>> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
>> at
>> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
>> at
>> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
>> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
>> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> 21/03/02 06:45:32 WARN WatchConnectionManager: Exec Failure
>> java.io.EOFException
>> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
>> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
>> at
>> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
>> at
>> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
>> at
>> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
>> at
>> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
>> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
>> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)
>> 21/03/02 06:50:33 WARN WatchConnectionManager: Exec Failure
>> java.io.EOFException
>> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
>> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
>> at
>> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
>> at
>> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
>> at
>> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
>> at
>> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
>> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
>> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.Threa

Re: Spark job crashing - Spark Structured Streaming with Kafka

2021-03-03 Thread Jungtaek Lim 
This looks to be coming from k8s, as WatchConnectionManager looks to be a
part of kubernetes-client. If that is intermittent, you may want to check
the health of k8s API server or network issues as well. Another bet would
be trying out 3.1.1 as Spark announces GA on supporting k8s from this
version.

On Tue, Mar 2, 2021 at 7:08 PM Sachit Murarka 
wrote:

> Hi Jungtaek,
>
> Please find full logs:
>
> java.io.EOFException
> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
> at
> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
> at
> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
> at
> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
> at
> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 21/03/02 06:40:31 WARN WatchConnectionManager: Exec Failure
> java.io.EOFException
> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
> at
> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
> at
> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
> at
> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
> at
> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 21/03/02 06:45:32 WARN WatchConnectionManager: Exec Failure
> java.io.EOFException
> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
> at
> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
> at
> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
> at
> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
> at
> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 21/03/02 06:50:33 WARN WatchConnectionManager: Exec Failure
> java.io.EOFException
> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
> at
> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
> at
> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
> at
> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
> at
> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
> at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 21/03/02 06:55:34 WARN WatchConnectionManager: Exec Failure
> java.io.EOFException
> at okio.RealBufferedSource.require(RealBufferedSource.java:61)
> at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
> at
> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
> at
> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
> at
> okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
> at
> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
> at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
> at okhttp3.internal.NamedRu

Re: Spark job crashing - Spark Structured Streaming with Kafka

2021-03-02 Thread Sachit Murarka 
Hi Jungtaek,

Please find full logs:

java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at
okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at
okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at
okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
21/03/02 06:40:31 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at
okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at
okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at
okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
21/03/02 06:45:32 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at
okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at
okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at
okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
21/03/02 06:50:33 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at
okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at
okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at
okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
21/03/02 06:55:34 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at
okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
at
okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at
okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
21/03/02 07:00:35 WARN WatchConnectionManager: Exec Failure
java.io.EOFException
at okio.RealBufferedSource.require(RealBufferedSource.java:61)
at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
at
okhttp3.int

Re: Spark job crashing - Spark Structured Streaming with Kafka

2021-03-02 Thread Jungtaek Lim 
I feel this quite lacks information. Full stack traces from
driver/executors are essential at least to determine what was happening.

On Tue, Mar 2, 2021 at 5:26 PM Sachit Murarka 
wrote:

> Hi All,
>
> My spark job is crashing (Structured stream) . Can anyone help please. I
> am using spark 3.0.1 with kubernetes.
>
> [ERROR] - StreamingQueryException Exception in  query.awaitTermination()
>   File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/streaming.py", line
> 103, in awaitTermination
> return self._jsq.awaitTermination()
>   File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",
> line 1304, in __call__
> return_value = get_return_value(
>   File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 134,
> in deco
> raise_from(converted)
>
> Kind Regards,
> Sachit Murarka
>