Re: Spark-shell doesn't see changes coming from Kafka topic

2016-12-05 Thread Otávio Carvalho
In the end, the mistake I made was that I forgot to setup the proper export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on the machine I was running the spark-shell. Nevertheless, thanks for answering, Tathagata Das. Otávio. 2016-12-01 17:36 GMT-02:00 Tathagata Das

Re: Spark-shell doesn't see changes coming from Kafka topic

2016-12-01 Thread Tathagata Das
Can you confirm the following? 1. Are you sending new data to the Kafka topic AFTER starting the streaming query? Since you have specified `*startingOffsets` *as* `latest`*, data needs to the topic after the query start for the query to receiver. 2. Are you able to read kafka data using Kafka's

Spark-shell doesn't see changes coming from Kafka topic

2016-12-01 Thread Otávio Carvalho
Hello hivemind, I am trying to connect my Spark 2.0.2 cluster to an Apache Kafka 0.10 cluster via spark-shell. The connection works fine, but it is not able to receive the messages published to the topic. It doesn't throw any error, but it is not able to retrieve any message (I am sure that