Re: How to convert Row to JSON in Java?

2017-09-11 Thread Riccardo Ferrari
I agree, Java tend to be pretty verbose unfortunately. You can check the "alternate" approach that should be more compact and readable. Should be something like: df.select(to_json(struct(col("*")).alias("value")) Of course to_json, struct and col are from the

Re: How to convert Row to JSON in Java?

2017-09-11 Thread kant kodali
getValuesMap is not very Java friendly. I have to do something like this String[] fieldNames = row.schema().fieldNames(); Seq fieldNamesSeq = JavaConverters.asScalaIteratorConverter(Arrays.asList(fieldNames).iterator()).asScala().toSeq(); String json = row.getValuesMap(fieldNamesSeq).toString();

Re: How to convert Row to JSON in Java?

2017-09-11 Thread Riccardo Ferrari
Hi Ayan, yup that works very well however I believe Kant's other mail "Queries with streaming sources must be executed with writeStream.start()" is adding more context. I think he is trying leverage on structured streaming and applying the rdd conversion to a streaming dataset is breaking the

Re: How to convert Row to JSON in Java?

2017-09-10 Thread Jean Georges Perrin
Sorry - more likely l700 save. jg > On Sep 10, 2017, at 20:56, Jean Georges Perrin wrote: > > Hey, > > I have a few examples https://github.com/jgperrin/net.jgp.labs.spark. I > recently worked on such problems, so there's definitely a solution there or > I'll be happy to

Re: How to convert Row to JSON in Java?

2017-09-10 Thread Jean Georges Perrin
Hey, I have a few examples https://github.com/jgperrin/net.jgp.labs.spark. I recently worked on such problems, so there's definitely a solution there or I'll be happy to write one for you. Look in l250 map... jg > On Sep 10, 2017, at 20:51, ayan guha wrote: > >

Re: How to convert Row to JSON in Java?

2017-09-10 Thread ayan guha
Sorry for side-line question, but for Python, isn't following the easiest: >>> import json >>>* df1 = df.rdd.map(lambda r: json.dumps(r.asDict()))* >>> df1.take(10) ['{"id": 1}', '{"id": 2}', '{"id": 3}', '{"id": 4}', '{"id": 5}'] On Mon, Sep 11, 2017 at 4:22 AM, Riccardo Ferrari

Re: How to convert Row to JSON in Java?

2017-09-10 Thread Riccardo Ferrari
Hi Kant, You can check the getValuesMap . I found this post useful, it is in Scala but should be a good starting point. An alternative

Re: How to convert Row to JSON in Java?

2017-09-09 Thread kant kodali
toJSON on Row object. On Sat, Sep 9, 2017 at 4:18 PM, Felix Cheung wrote: > toJSON on Dataset/DataFrame? > > -- > *From:* kant kodali > *Sent:* Saturday, September 9, 2017 4:15:49 PM > *To:* user @spark > *Subject:* How

Re: How to convert Row to JSON in Java?

2017-09-09 Thread Felix Cheung
toJSON on Dataset/DataFrame? From: kant kodali Sent: Saturday, September 9, 2017 4:15:49 PM To: user @spark Subject: How to convert Row to JSON in Java? Hi All, How to convert Row to JSON in Java? It would be nice to have .toJson() method

How to convert Row to JSON in Java?

2017-09-09 Thread kant kodali
Hi All, How to convert Row to JSON in Java? It would be nice to have .toJson() method in the Row class. Thanks, kant