Re: Is there a mutable dataframe spark structured streaming 2.3.0?
Quick Question regarding the same topic. If construct the data frame *df*
in the following way
key | value | topic| offset | partition | timestamp
foo1 | value1 | topic1 | ...
foo2 | value2 | topic2 | ...
what happens when I do df.writeStream().format("kafka").option("topic",
"hello_topic").start() ? Will it use the hello_topic specified in the
option or the topic from row ? Sure I don't need to specify topic as an
option if I have it in a row but I wonder what happens internally if this
is the case?
Thanks!
On Thu, Mar 22, 2018 at 7:48 AM, kant kodali wrote:
> Thanks all!
>
> On Thu, Mar 22, 2018 at 2:08 AM, Jorge Machado wrote:
>
>> DataFrames are not mutable.
>>
>> Jorge Machado
>>
>>
>> On 22 Mar 2018, at 10:07, Aakash Basu wrote:
>>
>> Hey,
>>
>> I faced the same issue a couple of days back, kindly go through the mail
>> chain with "*Multiple Kafka Spark Streaming Dataframe Join query*" as
>> subject, TD and Chris has cleared my doubts, it would help you too.
>>
>> Thanks,
>> Aakash.
>>
>> On Thu, Mar 22, 2018 at 7:50 AM, kant kodali wrote:
>>
>>> Hi All,
>>>
>>> Is there a mutable dataframe spark structured streaming 2.3.0? I am
>>> currently reading from Kafka and if I cannot parse the messages that I get
>>> from Kafka I want to write them to say some "dead_queue" topic.
>>>
>>> I wonder what is the best way to do this?
>>>
>>> Thanks!
>>>
>>
>>
>>
>
Re: Is there a mutable dataframe spark structured streaming 2.3.0?
Thanks all! On Thu, Mar 22, 2018 at 2:08 AM, Jorge Machado wrote: > DataFrames are not mutable. > > Jorge Machado > > > On 22 Mar 2018, at 10:07, Aakash Basu wrote: > > Hey, > > I faced the same issue a couple of days back, kindly go through the mail > chain with "*Multiple Kafka Spark Streaming Dataframe Join query*" as > subject, TD and Chris has cleared my doubts, it would help you too. > > Thanks, > Aakash. > > On Thu, Mar 22, 2018 at 7:50 AM, kant kodali wrote: > >> Hi All, >> >> Is there a mutable dataframe spark structured streaming 2.3.0? I am >> currently reading from Kafka and if I cannot parse the messages that I get >> from Kafka I want to write them to say some "dead_queue" topic. >> >> I wonder what is the best way to do this? >> >> Thanks! >> > > >
Re: Is there a mutable dataframe spark structured streaming 2.3.0?
DataFrames are not mutable. Jorge Machado > On 22 Mar 2018, at 10:07, Aakash Basu wrote: > > Hey, > > I faced the same issue a couple of days back, kindly go through the mail > chain with "Multiple Kafka Spark Streaming Dataframe Join query" as subject, > TD and Chris has cleared my doubts, it would help you too. > > Thanks, > Aakash. > > On Thu, Mar 22, 2018 at 7:50 AM, kant kodali <mailto:kanth...@gmail.com>> wrote: > Hi All, > > Is there a mutable dataframe spark structured streaming 2.3.0? I am currently > reading from Kafka and if I cannot parse the messages that I get from Kafka I > want to write them to say some "dead_queue" topic. > > I wonder what is the best way to do this? > > Thanks! >
Re: Is there a mutable dataframe spark structured streaming 2.3.0?
Hey, I faced the same issue a couple of days back, kindly go through the mail chain with "*Multiple Kafka Spark Streaming Dataframe Join query*" as subject, TD and Chris has cleared my doubts, it would help you too. Thanks, Aakash. On Thu, Mar 22, 2018 at 7:50 AM, kant kodali wrote: > Hi All, > > Is there a mutable dataframe spark structured streaming 2.3.0? I am > currently reading from Kafka and if I cannot parse the messages that I get > from Kafka I want to write them to say some "dead_queue" topic. > > I wonder what is the best way to do this? > > Thanks! >
Is there a mutable dataframe spark structured streaming 2.3.0?
Hi All, Is there a mutable dataframe spark structured streaming 2.3.0? I am currently reading from Kafka and if I cannot parse the messages that I get from Kafka I want to write them to say some "dead_queue" topic. I wonder what is the best way to do this? Thanks!
