Re: Issues: Generate JSON with null values in Spark 2.0.x

[Dongjin Lee]

Hi Chetan,

Sadly, you can not; Spark is configured to ignore the null values when
writing JSON. (check JacksonMessageWriter and find
JsonInclude.Include.NON_NULL from the code.) If you want that
functionality, it would be much better to file the problem to JIRA.

Best,
Dongjin

On Mon, Mar 20, 2017 at 4:44 PM, Chetan Khatri 
wrote:

> Exactly.
>
> On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee  wrote:
>
>> Hello Chetan,
>>
>> Could you post some code? If I understood correctly, you are trying to
>> save JSON like:
>>
>> {
>>   "first_name": "Dongjin",
>>   "last_name: null
>> }
>>
>> not in omitted form, like:
>>
>> {
>>   "first_name": "Dongjin"
>> }
>>
>> right?
>>
>> - Dongjin
>>
>> On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri <
>> chetan.opensou...@gmail.com> wrote:
>>
>>> Hello Dev / Users,
>>>
>>> I am working with PySpark Code migration to scala, with Python -
>>> Iterating Spark with dictionary and generating JSON with null is possible
>>> with json.dumps() which will be converted to SparkSQL[Row] but in scala how
>>> can we generate json will null values as a Dataframe ?
>>>
>>> Thanks.
>>>
>>
>>
>>
>> --
>> *Dongjin Lee*
>>
>>
>> *Software developer in Line+.So interested in massive-scale machine
>> learning.facebook: www.facebook.com/dongjin.lee.kr
>> linkedin: 
>> kr.linkedin.com/in/dongjinleekr
>> github:
>> github.com/dongjinleekr
>> twitter: www.twitter.com/dongjinleekr
>> *
>>
>
>


-- 
*Dongjin Lee*


*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
linkedin:
kr.linkedin.com/in/dongjinleekr
github:
github.com/dongjinleekr
twitter: www.twitter.com/dongjinleekr
*

Re: Issues: Generate JSON with null values in Spark 2.0.x

[Chetan Khatri]

Exactly.

On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee  wrote:

> Hello Chetan,
>
> Could you post some code? If I understood correctly, you are trying to
> save JSON like:
>
> {
>   "first_name": "Dongjin",
>   "last_name: null
> }
>
> not in omitted form, like:
>
> {
>   "first_name": "Dongjin"
> }
>
> right?
>
> - Dongjin
>
> On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri  > wrote:
>
>> Hello Dev / Users,
>>
>> I am working with PySpark Code migration to scala, with Python -
>> Iterating Spark with dictionary and generating JSON with null is possible
>> with json.dumps() which will be converted to SparkSQL[Row] but in scala how
>> can we generate json will null values as a Dataframe ?
>>
>> Thanks.
>>
>
>
>
> --
> *Dongjin Lee*
>
>
> *Software developer in Line+.So interested in massive-scale machine
> learning.facebook: www.facebook.com/dongjin.lee.kr
> linkedin: 
> kr.linkedin.com/in/dongjinleekr
> github:
> github.com/dongjinleekr
> twitter: www.twitter.com/dongjinleekr
> *
>

Re: Issues: Generate JSON with null values in Spark 2.0.x

[Dongjin Lee]

Hello Chetan,

Could you post some code? If I understood correctly, you are trying to save
JSON like:

{
  "first_name": "Dongjin",
  "last_name: null
}

not in omitted form, like:

{
  "first_name": "Dongjin"
}

right?

- Dongjin

On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri 
wrote:

> Hello Dev / Users,
>
> I am working with PySpark Code migration to scala, with Python - Iterating
> Spark with dictionary and generating JSON with null is possible with
> json.dumps() which will be converted to SparkSQL[Row] but in scala how can
> we generate json will null values as a Dataframe ?
>
> Thanks.
>



-- 
*Dongjin Lee*


*Software developer in Line+.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
linkedin:
kr.linkedin.com/in/dongjinleekr
github:
github.com/dongjinleekr
twitter: www.twitter.com/dongjinleekr
*

Issues: Generate JSON with null values in Spark 2.0.x

[Chetan Khatri]

Hello Dev / Users,

I am working with PySpark Code migration to scala, with Python - Iterating
Spark with dictionary and generating JSON with null is possible with
json.dumps() which will be converted to SparkSQL[Row] but in scala how can
we generate json will null values as a Dataframe ?

Thanks.