Spark Streaming handling Kafka exceptions

[Jean-Francois Gosselin]

How can I handle an error with Kafka with my DirectStream (network issue,
zookeeper or broker going down) ? For example when the consumer fails to
connect with Kafka (at startup) I only get a DEBUG log (not even an ERROR)
and no exception are thrown ...

I'm using Spark 2.1.1 and spark-streaming-kafka-0-10.

16:50:23.149 [ForkJoinPool-1-worker-5] DEBUG
o.a.kafka.common.network.Selector - Connection with localhost/127.0.0.1
disconnected
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:51)
at
org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:81)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:335)
at org.apache.kafka.common.network.Selector.poll(Selector.java:303)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:349)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:226)


Thanks

From C* to DataFrames with JSON

[Jean-Francois Gosselin]

Hi all,

I'm struggling (Spark / Scala newbie) to create a DataFrame from a C* table
but also create a DataFrame from column with json .

e.g. From C* table

| id | jsonData  |
==
| 1 |  {"a": "123", "b": "xyz" } |
+--+---+
| 2 |  {"a": "3", "b": "bar" } |


to Spark DataFrame:

| id |  a   |  b   |
===
| 1 | 123 | xyz |
+--+--+-+
| 2 | 3 | bar |


I'm using Spark 1.6 .

Thanks


JF