Re: How to read from multiple kafka topics using structured streaming (spark 2.2.0)?

2017-09-19 Thread Jacek Laskowski 
Hi,

Cody's right.

subscribe - Topic subscription strategy that accepts topic names as a
comma-separated string, e.g. topic1,topic2,topic3 [1]

subscribepattern - Topic subscription strategy that uses Java’s
java.util.regex.Pattern for the topic subscription regex pattern of topics
to subscribe to, e.g. topic\d [2]

[1]
https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSource.html#subscribe
[2]
https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSource.html#subscribepattern

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
Spark Structured Streaming (Apache Spark 2.2+)
https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

On Tue, Sep 19, 2017 at 10:34 PM, Cody Koeninger  wrote:

> You should be able to pass a comma separated string of topics to
> subscribe.  subscribePattern isn't necessary
>
>
>
> On Tue, Sep 19, 2017 at 2:54 PM, kant kodali  wrote:
> > got it! Sorry.
> >
> > On Tue, Sep 19, 2017 at 12:52 PM, Jacek Laskowski 
> wrote:
> >>
> >> Hi,
> >>
> >> Use subscribepattern
> >>
> >> You haven't googled well enough -->
> >> https://jaceklaskowski.gitbooks.io/spark-structured-
> streaming/spark-sql-streaming-KafkaSource.html
> >> :)
> >>
> >> Pozdrawiam,
> >> Jacek Laskowski
> >> 
> >> https://about.me/JacekLaskowski
> >> Spark Structured Streaming (Apache Spark 2.2+)
> >> https://bit.ly/spark-structured-streaming
> >> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
> >> Follow me at https://twitter.com/jaceklaskowski
> >>
> >> On Tue, Sep 19, 2017 at 9:50 PM, kant kodali 
> wrote:
> >>>
> >>> HI All,
> >>>
> >>> I am wondering How to read from multiple kafka topics using structured
> >>> streaming (code below)? I googled prior to asking this question and I
> see
> >>> responses related to Dstreams but not structured streams. Is it
> possible to
> >>> read multiple topics using the same spark structured stream?
> >>>
> >>> sparkSession.readStream()
> >>> .format("kafka")
> >>> .option("kafka.bootstrap.servers", "localhost:9092")
> >>> .option("subscribe", "hello1")
> >>> .option("startingOffsets", "earliest")
> >>> .option("failOnDataLoss", "false")
> >>> .load();
> >>>
> >>>
> >>> Thanks!
> >>
> >>
> >
>

Re: How to read from multiple kafka topics using structured streaming (spark 2.2.0)?

2017-09-19 Thread Cody Koeninger 
You should be able to pass a comma separated string of topics to
subscribe.  subscribePattern isn't necessary



On Tue, Sep 19, 2017 at 2:54 PM, kant kodali  wrote:
> got it! Sorry.
>
> On Tue, Sep 19, 2017 at 12:52 PM, Jacek Laskowski  wrote:
>>
>> Hi,
>>
>> Use subscribepattern
>>
>> You haven't googled well enough -->
>> https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSource.html
>> :)
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> Spark Structured Streaming (Apache Spark 2.2+)
>> https://bit.ly/spark-structured-streaming
>> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> On Tue, Sep 19, 2017 at 9:50 PM, kant kodali  wrote:
>>>
>>> HI All,
>>>
>>> I am wondering How to read from multiple kafka topics using structured
>>> streaming (code below)? I googled prior to asking this question and I see
>>> responses related to Dstreams but not structured streams. Is it possible to
>>> read multiple topics using the same spark structured stream?
>>>
>>> sparkSession.readStream()
>>> .format("kafka")
>>> .option("kafka.bootstrap.servers", "localhost:9092")
>>> .option("subscribe", "hello1")
>>> .option("startingOffsets", "earliest")
>>> .option("failOnDataLoss", "false")
>>> .load();
>>>
>>>
>>> Thanks!
>>
>>
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: How to read from multiple kafka topics using structured streaming (spark 2.2.0)?

2017-09-19 Thread kant kodali 
got it! Sorry.

On Tue, Sep 19, 2017 at 12:52 PM, Jacek Laskowski  wrote:

> Hi,
>
> Use subscribepattern
>
> You haven't googled well enough --> https://jaceklaskowski.
> gitbooks.io/spark-structured-streaming/spark-sql-streaming-
> KafkaSource.html :)
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> Spark Structured Streaming (Apache Spark 2.2+) https://bit.ly/spark-
> structured-streaming
> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> On Tue, Sep 19, 2017 at 9:50 PM, kant kodali  wrote:
>
>> HI All,
>>
>> I am wondering How to read from multiple kafka topics using structured
>> streaming (code below)? I googled prior to asking this question and I see
>> responses related to Dstreams but not structured streams. Is it possible to
>> read multiple topics using the same spark structured stream?
>>
>> sparkSession.readStream()
>> .format("kafka")
>> .option("kafka.bootstrap.servers", "localhost:9092")
>> .option("subscribe", "hello1")
>> .option("startingOffsets", "earliest")
>> .option("failOnDataLoss", "false")
>> .load();
>>
>>
>> Thanks!
>>
>>
>

Re: How to read from multiple kafka topics using structured streaming (spark 2.2.0)?

2017-09-19 Thread Jacek Laskowski 
Hi,

Use subscribepattern

You haven't googled well enough -->
https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-KafkaSource.html
:)

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
Spark Structured Streaming (Apache Spark 2.2+)
https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

On Tue, Sep 19, 2017 at 9:50 PM, kant kodali  wrote:

> HI All,
>
> I am wondering How to read from multiple kafka topics using structured
> streaming (code below)? I googled prior to asking this question and I see
> responses related to Dstreams but not structured streams. Is it possible to
> read multiple topics using the same spark structured stream?
>
> sparkSession.readStream()
> .format("kafka")
> .option("kafka.bootstrap.servers", "localhost:9092")
> .option("subscribe", "hello1")
> .option("startingOffsets", "earliest")
> .option("failOnDataLoss", "false")
> .load();
>
>
> Thanks!
>
>

How to read from multiple kafka topics using structured streaming (spark 2.2.0)?

2017-09-19 Thread kant kodali 
HI All,

I am wondering How to read from multiple kafka topics using structured
streaming (code below)? I googled prior to asking this question and I see
responses related to Dstreams but not structured streams. Is it possible to
read multiple topics using the same spark structured stream?

sparkSession.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "hello1")
.option("startingOffsets", "earliest")
.option("failOnDataLoss", "false")
.load();


Thanks!